Sep 10 20:55:51.857349 ip-10-0-134-6 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 20:55:51.857361 ip-10-0-134-6 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 20:55:51.857368 ip-10-0-134-6 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 20:55:51.857619 ip-10-0-134-6 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 20:56:02.099076 ip-10-0-134-6 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 20:56:02.099091 ip-10-0-134-6 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot edb10ce628a44ae49abbc4c32ef09f47 --
Sep 10 20:57:38.940486 ip-10-0-134-6 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 20:57:39.336869 ip-10-0-134-6 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:57:39.336869 ip-10-0-134-6 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 20:57:39.336869 ip-10-0-134-6 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:57:39.336869 ip-10-0-134-6 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 20:57:39.336869 ip-10-0-134-6 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:57:39.337949 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.337863    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 20:57:39.341214 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341177    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:57:39.341214 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341194    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:57:39.341214 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341214    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:57:39.341214 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341218    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341223    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341228    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341231    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341235    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341239    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341243    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341246    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341250    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341254    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341258    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341262    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341265    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341269    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341273    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341277    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341281    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341285    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341289    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341293    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:57:39.341455 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341297    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341301    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341312    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341317    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341321    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341325    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341329    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341333    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341337    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341342    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341349    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341354    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341360    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341366    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341374    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341379    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341384    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341389    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341393    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:57:39.342272 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341398    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341403    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341407    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341412    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341416    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341420    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341424    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341429    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341433    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341437    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341441    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341445    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341450    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341455    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341461    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341465    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341472    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341477    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341481    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:57:39.342737 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341486    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341491    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341496    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341500    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341504    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341508    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341513    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341516    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341521    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341526    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341530    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341534    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341539    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341543    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341547    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341552    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341556    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341560    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341565    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341569    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:57:39.343317 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341574    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:57:39.343805 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341578    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:57:39.343805 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341582    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:57:39.343805 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341587    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:57:39.343805 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341591    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:57:39.343805 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341595    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:57:39.343805 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341599    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:57:39.343805 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341605    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:57:39.343805 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341609    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:57:39.343805 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341613    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:57:39.343805 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341617    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:57:39.343805 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341622    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:57:39.343805 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341626    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:57:39.343805 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341631    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:57:39.343805 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341635    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:57:39.343805 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341639    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:57:39.343805 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341643    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:57:39.343805 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.341648    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:57:39.345424 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345407    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:57:39.345424 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345421    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:57:39.345424 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345426    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:57:39.345424 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345431    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345435    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345440    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345444    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345448    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345452    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345457    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345462    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345466    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345470    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345474    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345478    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345482    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345486    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345491    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345495    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345500    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345504    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345509    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345512    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:57:39.345660 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345517    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345520    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345524    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345529    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345533    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345537    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345541    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345546    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345550    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345554    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345558    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345562    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345566    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345570    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345576    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345581    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345585    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345589    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345593    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345598    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:57:39.346537 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345603    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345607    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345611    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345616    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345620    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345626    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345630    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345634    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345638    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345642    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345647    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345651    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345655    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345659    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345663    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345667    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345672    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345676    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345680    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345684    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:57:39.347423 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345688    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345692    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345697    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345701    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345705    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345709    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345714    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345719    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345723    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345727    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345731    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345735    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345740    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345745    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345753    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345759    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345764    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345768    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345773    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:57:39.348313 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345780    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345786    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345791    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345796    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345800    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345805    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345809    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345813    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345817    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345821    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345826    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345830    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345834    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345849    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345855    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345860    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.345864    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346488    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346497    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:57:39.348913 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346502    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346507    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346512    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346516    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346520    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346525    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346530    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346534    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346539    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346542    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346546    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346550    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346554    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346558    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346562    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346566    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346570    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346574    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346579    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346583    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:57:39.349483 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346587    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346592    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346595    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346600    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346605    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346609    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346613    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346617    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346621    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346626    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346630    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346635    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346639    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346644    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346648    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346652    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346657    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346662    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346666    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:57:39.349988 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346671    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346678    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346685    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346692    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346697    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346702    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346707    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346712    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346716    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346720    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346724    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346728    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346732    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346737    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346741    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346745    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346752    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346757    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346761    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346765    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:57:39.350498 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346770    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346775    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346779    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346783    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346788    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346793    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346797    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346801    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346805    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346809    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346813    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346817    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346821    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346826    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346830    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346834    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346839    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346843    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346847    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346851    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:57:39.351082 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346855    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346859    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346863    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346867    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346871    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346875    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346881    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346887    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346893    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346898    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346902    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346906    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346911    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346915    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346919    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346923    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346927    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.346932    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347033    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347083    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 20:57:39.351748 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347092    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347098    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347104    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347109    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347116    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347122    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347129    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347134    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347140    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347145    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347150    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347154    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347159    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347164    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347169    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347174    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347183    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347188    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347193    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347216    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347224    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347230    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347235    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 20:57:39.352323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347240    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347246    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347250    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347255    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347259    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347265    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347269    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347276    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347281    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347286    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347290    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347295    2581 flags.go:64] FLAG: --enable-server="true"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347300    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347307    2581 flags.go:64] FLAG: --event-burst="100"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347312    2581 flags.go:64] FLAG: --event-qps="50"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347317    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347323    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347328    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347334    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347338    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347343    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347348    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347353    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347358    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347362    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 20:57:39.352965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347368    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347373    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347378    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347383    2581 flags.go:64] FLAG: --feature-gates=""
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347391    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347396    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347401    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347406    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347411    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347416    2581 flags.go:64] FLAG: --help="false"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347421    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347426    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347431    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347435    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347440    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347446    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347450    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347455    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347460    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347465    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347469    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347475    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347479    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347484    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 20:57:39.353602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347490    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347495    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347500    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347505    2581 flags.go:64] FLAG: --lock-file=""
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347509    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347514    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347519    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347527    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347534    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347538    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347543    2581 flags.go:64] FLAG: --logging-format="text"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347547    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347555    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347559    2581 flags.go:64] FLAG: --manifest-url=""
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347564    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347571    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347576    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347582    2581 flags.go:64] FLAG: --max-pods="110"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347587    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347592    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347597    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347602    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347606    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347611    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347616    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 20:57:39.354254 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347627    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347632    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347637    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347642    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347646    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347655    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347659    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347664    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347669    2581 flags.go:64] FLAG: --port="10250"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347675    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347679    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e71f273aaf96557b"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347685    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347690    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347694    2581 flags.go:64] FLAG: --register-node="true"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347699    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347704    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347711    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347716    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347720    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347728    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347733    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347738    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347743    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347748    2581 flags.go:64] FLAG: --runonce="false"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347752    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 20:57:39.354872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347758    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347763    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347767    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347772    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347777    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347782    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347787    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347792    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347796    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347801    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347806    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347811    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347815    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347820    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347829    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347833    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347838    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347845    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347850    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347855    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347859    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347864    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347869    2581 flags.go:64] FLAG: --v="2"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347877    2581 flags.go:64] FLAG: --version="false"
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347883    2581 flags.go:64] FLAG: --vmodule=""
Sep 10 20:57:39.355504 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347891    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.347899    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348112    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348119    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348125    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348132    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348137    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348142    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348147    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348151    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348156    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348160    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348164    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348168    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348173    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348177    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348181    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348186    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348190    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:57:39.356274 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348194    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348214    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348218    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348223    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348227    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348232    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348236    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348240    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348244    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348250    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348256    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348263    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348269    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348274    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348282    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348287    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348292    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348297    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348301    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348305    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:57:39.356755 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348309    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348314    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348318    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348322    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348326    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348330    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348334    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348338    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348342    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348347    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348351    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348355    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348359    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348364    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348368    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348372    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348376    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348380    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348385    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348389    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:57:39.357266 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348394    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348398    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348402    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348408    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348412    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348416    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348421    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348426    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348430    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348434    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348438    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348442    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348447    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348451    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348455    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348459    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348463    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348468    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348472    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:57:39.357748 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348476    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348480    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348484    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348488    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348492    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348497    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348501    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348505    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348509    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348513    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348517    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348521    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348529    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348535    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348540    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348544    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348550    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348554    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348559    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348563    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:57:39.358227 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348568    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:57:39.358720 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348571    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:57:39.358720 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.348576    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:57:39.358720 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.349333    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:57:39.358720 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.357518    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 20:57:39.358720 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.357535    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 20:57:39.358720 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.358424    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 20:57:39.358720 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.358434    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 20:57:39.358720 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358500    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:57:39.358720 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358505    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:57:39.358720 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358509    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:57:39.358720 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358512    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:57:39.358720 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358515    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:57:39.358720 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358518    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:57:39.358720 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358521    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:57:39.358720 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358523    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358526    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358529    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358531    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358534    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358537    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358539    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358542    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358545    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358547    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358550    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358552    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358555    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358557    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358560    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358562    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358565    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358567    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358570    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358572    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:57:39.359102 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358575    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358577    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358586    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358589    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358592    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358595    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358598    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358600    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358603    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358607    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358611    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358614    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358618    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358621    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358624    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358627    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358630    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358633    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358636    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:57:39.359672 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358638    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358641    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358643    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358646    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358648    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358651    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358653    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358656    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358659    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358661    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358664    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358666    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358670    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358674    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358677    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358680    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358688    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358692    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358694    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358697    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:57:39.360134 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358699    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358702    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358705    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358707    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358710    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358712    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358715    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358717    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358720    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358723    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358725    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358728    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358731    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358734    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358736    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358739    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358741    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358744    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358746    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358749    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:57:39.360631 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358751    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:57:39.361094 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358754    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:57:39.361094 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358756    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:57:39.361094 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358759    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:57:39.361094 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358761    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:57:39.361094 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358764    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:57:39.361094 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358767    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:57:39.361094 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358770    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:57:39.361094 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358772    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:57:39.361094 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358780    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:57:39.361094 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358783    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:57:39.361094 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358786    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:57:39.361094 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358789    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:57:39.361094 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.358794    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:57:39.361094 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358949    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:57:39.361094 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358954    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358957    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358960    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358964    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358968    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358971    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358974    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358977    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358980    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358983    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358986    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358989    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358992    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358995    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.358998    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359001    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359004    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359006    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359009    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:57:39.361489 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359012    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359015    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359017    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359020    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359023    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359026    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359028    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359031    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359039    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359043    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359047    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359050    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359052    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359055    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359058    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359060    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359063    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359065    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359068    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359070    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:57:39.361942 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359073    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359076    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359078    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359081    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359083    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359086    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359088    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359091    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359093    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359096    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359098    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359101    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359103    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359106    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359108    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359111    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359114    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359117    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359120    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359122    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:57:39.362446 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359125    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359132    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359135    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359138    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359140    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359143    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359145    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359147    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359150    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359153    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359155    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359157    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359160    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359162    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359165    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359167    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359170    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359172    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359175    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359177    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:57:39.362932 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359180    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359182    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359185    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359187    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359189    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359192    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359195    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359212    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359217    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359221    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359225    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359228    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359231    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359234    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359242    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359245    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359247    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359250    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:57:39.363440 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:39.359253    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:57:39.363874 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.359257    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:57:39.363874 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.359379    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 20:57:39.363874 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.361775    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 20:57:39.363874 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.362854    2581 server.go:1019] "Starting client certificate rotation"
Sep 10 20:57:39.363874 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.362945    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 20:57:39.363874 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.362980    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 20:57:39.365829 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.365810    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 20:57:39.370986 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.370966    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 20:57:39.381561 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.381542    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 10 20:57:39.385741 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.385728    2581 log.go:25] "Validated CRI v1 image API"
Sep 10 20:57:39.386613 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.386599    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 20:57:39.392144 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.392123    2581 fs.go:135] Filesystem UUIDs: map[06d98a38-b01b-46d4-a872-13eb0ac68400:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8d14919d-df85-4fd3-89ee-eebed9c1c8c4:/dev/nvme0n1p4]
Sep 10 20:57:39.392225 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.392144    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 20:57:39.397328 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.397184    2581 manager.go:216] Machine: {Timestamp:2026-09-10 20:57:39.395690855 +0000 UTC m=+0.371350894 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099057 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2719c21f76b909f45f9990641e07a0 SystemUUID:ec2719c2-1f76-b909-f45f-9990641e07a0 BootID:edb10ce6-28a4-4ae4-9abb-c4c32ef09f47 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a3:4b:19:68:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a3:4b:19:68:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:19:ef:31:db:cc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 20:57:39.397328 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.397301    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 20:57:39.397499 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.397375    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 20:57:39.397499 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.397458    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 20:57:39.401117 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.401089    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 20:57:39.401292 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.401121    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-6.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 20:57:39.402052 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.402040    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 20:57:39.402115 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.402060    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 20:57:39.402115 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.402072    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 20:57:39.402949 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.402936    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 20:57:39.404256 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.404244    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 20:57:39.408848 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.408834    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 20:57:39.408980 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.408969    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 20:57:39.409047 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.409009    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 20:57:39.409047 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.409022    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 20:57:39.409047 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.409042    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 20:57:39.409177 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.409056    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 10 20:57:39.409177 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.409069    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 20:57:39.409464 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.409451    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 20:57:39.409527 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.409474    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 20:57:39.411674 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.411660    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 20:57:39.414518 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.414505    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 20:57:39.414601 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.414536    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 20:57:39.416496 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.416485    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 20:57:39.416542 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.416500    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 20:57:39.416542 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.416506    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 20:57:39.416542 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.416511    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 20:57:39.416542 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.416520    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 20:57:39.416542 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.416526    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 20:57:39.416542 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.416539    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 20:57:39.416542 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.416545    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 20:57:39.416730 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.416551    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 20:57:39.416730 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.416557    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 20:57:39.416730 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.416570    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 20:57:39.416730 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.416576    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 20:57:39.418328 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.418317    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 20:57:39.418328 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.418328    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 20:57:39.420084 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.420060    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 20:57:39.420146 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.420065    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 20:57:39.422045 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.422034    2581 server.go:1268] "Started kubelet"
Sep 10 20:57:39.422147 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.422105    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 20:57:39.422195 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.422165    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 20:57:39.422195 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.422144    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 20:57:39.426830 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.426730    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 20:57:39.427068 ip-10-0-134-6 systemd[1]: Started Kubernetes Kubelet.
Sep 10 20:57:39.429967 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.429950    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 20:57:39.433944 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.433923    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-6.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 20:57:39.434046 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.434015    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 20:57:39.435142 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.434054    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-6.ec2.internal.18d411323bc9296c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-6.ec2.internal,UID:ip-10-0-134-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-6.ec2.internal,},FirstTimestamp:2026-09-10 20:57:39.422005612 +0000 UTC m=+0.397665650,LastTimestamp:2026-09-10 20:57:39.422005612 +0000 UTC m=+0.397665650,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-6.ec2.internal,}"
Sep 10 20:57:39.436936 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.436920    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 20:57:39.437679 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.437660    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 20:57:39.437817 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.437803    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-6.ec2.internal\" not found"
Sep 10 20:57:39.438402 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.438388    2581 factory.go:55] Registering systemd factory
Sep 10 20:57:39.438402 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.438404    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 10 20:57:39.438516 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.438464    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 20:57:39.438516 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.438488    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 20:57:39.438593 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.438464    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 20:57:39.438593 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.438590    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 20:57:39.438674 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.438599    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 20:57:39.438674 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.438622    2581 factory.go:153] Registering CRI-O factory
Sep 10 20:57:39.438674 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.438637    2581 factory.go:223] Registration of the crio container factory successfully
Sep 10 20:57:39.438766 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.438687    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 20:57:39.438766 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.438706    2581 factory.go:103] Registering Raw factory
Sep 10 20:57:39.438766 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.438723    2581 manager.go:1196] Started watching for new ooms in manager
Sep 10 20:57:39.439085 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.439072    2581 manager.go:349] Starting recovery of all containers
Sep 10 20:57:39.440001 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.439957    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 20:57:39.440127 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.440107    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 20:57:39.446018 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.445899    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 20:57:39.448669 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.448653    2581 manager.go:354] Recovery completed
Sep 10 20:57:39.453353 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.453341    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:57:39.455430 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.455402    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:57:39.455493 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.455444    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:57:39.455493 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.455459    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:57:39.456151 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.456132    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 20:57:39.456151 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.456149    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 20:57:39.456350 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.456156    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:57:39.456350 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.456168    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 20:57:39.456998 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.456892    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:57:39.456998 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.456927    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:57:39.456998 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.456941    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:57:39.458702 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.458681    2581 policy_none.go:49] "None policy: Start"
Sep 10 20:57:39.458807 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.458707    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 20:57:39.458807 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.458725    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 20:57:39.460818 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.460802    2581 policy_none.go:47] "Start"
Sep 10 20:57:39.461801 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.461738    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-6.ec2.internal.18d411323dc72506  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-6.ec2.internal,UID:ip-10-0-134-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-6.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-6.ec2.internal,},FirstTimestamp:2026-09-10 20:57:39.455427846 +0000 UTC m=+0.431087892,LastTimestamp:2026-09-10 20:57:39.455427846 +0000 UTC m=+0.431087892,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-6.ec2.internal,}"
Sep 10 20:57:39.466392 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.466318    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-6.ec2.internal.18d411323dc77eeb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-6.ec2.internal,UID:ip-10-0-134-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-6.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-6.ec2.internal,},FirstTimestamp:2026-09-10 20:57:39.455450859 +0000 UTC m=+0.431110898,LastTimestamp:2026-09-10 20:57:39.455450859 +0000 UTC m=+0.431110898,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-6.ec2.internal,}"
Sep 10 20:57:39.471624 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.471560    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-6.ec2.internal.18d411323dc7b271  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-6.ec2.internal,UID:ip-10-0-134-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-6.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-6.ec2.internal,},FirstTimestamp:2026-09-10 20:57:39.455464049 +0000 UTC m=+0.431124087,LastTimestamp:2026-09-10 20:57:39.455464049 +0000 UTC m=+0.431124087,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-6.ec2.internal,}"
Sep 10 20:57:39.477288 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.477231    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-6.ec2.internal.18d411323dc72506\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-6.ec2.internal.18d411323dc72506  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-6.ec2.internal,UID:ip-10-0-134-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-6.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-6.ec2.internal,},FirstTimestamp:2026-09-10 20:57:39.455427846 +0000 UTC m=+0.431087892,LastTimestamp:2026-09-10 20:57:39.45691014 +0000 UTC m=+0.432570180,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-6.ec2.internal,}"
Sep 10 20:57:39.486119 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.486064    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-6.ec2.internal.18d411323dc77eeb\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-6.ec2.internal.18d411323dc77eeb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-6.ec2.internal,UID:ip-10-0-134-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-6.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-6.ec2.internal,},FirstTimestamp:2026-09-10 20:57:39.455450859 +0000 UTC m=+0.431110898,LastTimestamp:2026-09-10 20:57:39.456934168 +0000 UTC m=+0.432594208,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-6.ec2.internal,}"
Sep 10 20:57:39.509994 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.488912    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-6.ec2.internal.18d411323dc7b271\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-6.ec2.internal.18d411323dc7b271  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-6.ec2.internal,UID:ip-10-0-134-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-6.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-6.ec2.internal,},FirstTimestamp:2026-09-10 20:57:39.455464049 +0000 UTC m=+0.431124087,LastTimestamp:2026-09-10 20:57:39.456947146 +0000 UTC m=+0.432607186,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-6.ec2.internal,}"
Sep 10 20:57:39.509994 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.491741    2581 manager.go:337] "Starting Device Plugin manager"
Sep 10 20:57:39.509994 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.491770    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 20:57:39.509994 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.491779    2581 server.go:87] "Starting device plugin registration server"
Sep 10 20:57:39.509994 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.492090    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 20:57:39.509994 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.492102    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 20:57:39.509994 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.492245    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 20:57:39.509994 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.492318    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 20:57:39.509994 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.492327    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 20:57:39.509994 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.492860    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 20:57:39.509994 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.492906    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-6.ec2.internal\" not found"
Sep 10 20:57:39.509994 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.499024    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-6.ec2.internal.18d41132401102c2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-6.ec2.internal,UID:ip-10-0-134-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-134-6.ec2.internal,},FirstTimestamp:2026-09-10 20:57:39.49382317 +0000 UTC m=+0.469483199,LastTimestamp:2026-09-10 20:57:39.49382317 +0000 UTC m=+0.469483199,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-6.ec2.internal,}"
Sep 10 20:57:39.567189 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.567162    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 20:57:39.567309 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.567225    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 20:57:39.567309 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.567249    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 20:57:39.567309 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.567286    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 20:57:39.570475 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.570451    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 10 20:57:39.586819 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.586802    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dp829"
Sep 10 20:57:39.592905 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.592851    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:57:39.593812 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.593798    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:57:39.593869 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.593827    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:57:39.593869 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.593840    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:57:39.593869 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.593862    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.601043 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.601028    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dp829"
Sep 10 20:57:39.605414 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.605332    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-6.ec2.internal.18d411323dc72506\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-6.ec2.internal.18d411323dc72506  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-6.ec2.internal,UID:ip-10-0-134-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-6.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-6.ec2.internal,},FirstTimestamp:2026-09-10 20:57:39.455427846 +0000 UTC m=+0.431087892,LastTimestamp:2026-09-10 20:57:39.593810825 +0000 UTC m=+0.569470861,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-6.ec2.internal,}"
Sep 10 20:57:39.619349 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.619332    2581 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-134-6.ec2.internal\" not found" node="ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.645752 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.645731    2581 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-6.ec2.internal\" not found" node="ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.667930 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.667911    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-6.ec2.internal"]
Sep 10 20:57:39.668004 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.667985    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:57:39.668803 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.668780    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:57:39.668892 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.668810    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:57:39.668892 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.668824    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:57:39.671161 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.671147    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:57:39.671307 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.671297    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.671361 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.671323    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:57:39.671848 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.671825    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:57:39.671848 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.671833    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:57:39.671963 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.671854    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:57:39.671963 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.671869    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:57:39.671963 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.671857    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:57:39.671963 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.671951    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:57:39.674178 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.674161    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.674275 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.674190    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:57:39.674862 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.674848    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:57:39.674917 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.674873    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:57:39.674917 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.674883    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:57:39.700818 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.700800    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-6.ec2.internal\" not found" node="ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.705059 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.705045    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-6.ec2.internal\" not found" node="ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.740461 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.740446    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fdce11d1461a1b096b934b218406a2b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal\" (UID: \"fdce11d1461a1b096b934b218406a2b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.740534 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.740469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdce11d1461a1b096b934b218406a2b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal\" (UID: \"fdce11d1461a1b096b934b218406a2b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.740534 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.740486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb31595c699a42d1829717a278752f0e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-6.ec2.internal\" (UID: \"fb31595c699a42d1829717a278752f0e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.819656 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.819636    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:57:39.820459 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.820439    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:57:39.820543 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.820465    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:57:39.820543 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.820475    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:57:39.820543 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.820499    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.827496 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.827482    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.827548 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.827501    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-6.ec2.internal\": node \"ip-10-0-134-6.ec2.internal\" not found"
Sep 10 20:57:39.840721 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.840701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fdce11d1461a1b096b934b218406a2b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal\" (UID: \"fdce11d1461a1b096b934b218406a2b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.840814 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.840731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdce11d1461a1b096b934b218406a2b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal\" (UID: \"fdce11d1461a1b096b934b218406a2b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.840814 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.840750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb31595c699a42d1829717a278752f0e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-6.ec2.internal\" (UID: \"fb31595c699a42d1829717a278752f0e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.840814 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.840778    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fdce11d1461a1b096b934b218406a2b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal\" (UID: \"fdce11d1461a1b096b934b218406a2b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.840814 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.840784    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb31595c699a42d1829717a278752f0e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-6.ec2.internal\" (UID: \"fb31595c699a42d1829717a278752f0e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.840947 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:39.840811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdce11d1461a1b096b934b218406a2b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal\" (UID: \"fdce11d1461a1b096b934b218406a2b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal"
Sep 10 20:57:39.919147 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:39.919092    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-6.ec2.internal\" not found"
Sep 10 20:57:40.003512 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:40.003486    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal"
Sep 10 20:57:40.008170 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:40.008154    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-6.ec2.internal"
Sep 10 20:57:40.019717 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:40.019697    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-6.ec2.internal\" not found"
Sep 10 20:57:40.120189 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:40.120166    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-6.ec2.internal\" not found"
Sep 10 20:57:40.223625 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:40.223599    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-6.ec2.internal\" not found"
Sep 10 20:57:40.285646 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:40.285624    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:57:40.324683 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:40.324655    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-6.ec2.internal\" not found"
Sep 10 20:57:40.363539 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:40.363512    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 20:57:40.363961 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:40.363649    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 20:57:40.425668 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:40.425607    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-6.ec2.internal\" not found"
Sep 10 20:57:40.451867 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:40.450553    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:57:40.467492 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:40.467473    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:57:40.481247 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:40.481225    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:57:40.483133 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:40.483117    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:57:40.483205 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:40.483147    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:57:40.483205 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:40.483158    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:57:40.526182 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:40.526147    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-6.ec2.internal\" not found"
Sep 10 20:57:40.605468 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:40.605428    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 20:52:40 +0000 UTC" deadline="2028-06-09 04:06:32.956745796 +0000 UTC"
Sep 10 20:57:40.605468 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:40.605464    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15295h8m52.351286362s"
Sep 10 20:57:40.626637 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:40.626606    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-6.ec2.internal\" not found"
Sep 10 20:57:40.661448 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:40.661408    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb31595c699a42d1829717a278752f0e.slice/crio-f744090471859bc84dc5e641fd2a88cdf27fed86f569c28d44f172fb3670c127 WatchSource:0}: Error finding container f744090471859bc84dc5e641fd2a88cdf27fed86f569c28d44f172fb3670c127: Status 404 returned error can't find the container with id f744090471859bc84dc5e641fd2a88cdf27fed86f569c28d44f172fb3670c127
Sep 10 20:57:40.662066 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:40.662044    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdce11d1461a1b096b934b218406a2b8.slice/crio-0ecfa2de739713e4a1303c7a55246901bcd3aa08605966f16843d8a6eb919f3d WatchSource:0}: Error finding container 0ecfa2de739713e4a1303c7a55246901bcd3aa08605966f16843d8a6eb919f3d: Status 404 returned error can't find the container with id 0ecfa2de739713e4a1303c7a55246901bcd3aa08605966f16843d8a6eb919f3d
Sep 10 20:57:40.666472 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:40.666455    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:57:40.729659 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:40.729599    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-6.ec2.internal\" not found"
Sep 10 20:57:40.832002 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:40.831969    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-6.ec2.internal\" not found"
Sep 10 20:57:40.933270 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:40.933239    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-6.ec2.internal\" not found"
Sep 10 20:57:40.954334 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:40.954313    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:57:41.038096 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.038029    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-6.ec2.internal"
Sep 10 20:57:41.049479 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.049457    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 20:57:41.050364 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.050350    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal"
Sep 10 20:57:41.066378 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.066358    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 20:57:41.413487 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.413361    2581 apiserver.go:52] "Watching apiserver"
Sep 10 20:57:41.422284 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.422145    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 20:57:41.423077 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.423041    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-ps8fr","openshift-image-registry/node-ca-ccgh9","openshift-multus/multus-additional-cni-plugins-qg95l","openshift-network-diagnostics/network-check-target-lpwwj","openshift-ovn-kubernetes/ovnkube-node-m777h","kube-system/global-pull-secret-syncer-29b5p","kube-system/konnectivity-agent-v6rlb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87","openshift-cluster-node-tuning-operator/tuned-qp845","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal","openshift-multus/multus-j5bg8","openshift-multus/network-metrics-daemon-4r2gd","openshift-network-operator/iptables-alerter-dnjhl","kube-system/kube-apiserver-proxy-ip-10-0-134-6.ec2.internal"]
Sep 10 20:57:41.425796 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.425766    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v6rlb"
Sep 10 20:57:41.428829 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.428314    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ps8fr"
Sep 10 20:57:41.429326 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.429086    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 20:57:41.429326 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.429121    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4k722\""
Sep 10 20:57:41.429490 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.429433    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 20:57:41.431765 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.430827    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.431765 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.431589    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 20:57:41.432507 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.432262    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 20:57:41.432606 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.432437    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5b689\""
Sep 10 20:57:41.433251 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.432971    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:57:41.433251 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:41.433043    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpwwj" podUID="97ae132c-459a-4cba-bb9f-822279e84225"
Sep 10 20:57:41.433528 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.433508    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 20:57:41.433602 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.433551    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jjq2l\""
Sep 10 20:57:41.433662 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.433609    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 20:57:41.433753 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.433735    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 20:57:41.435672 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.435652    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.438399 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.437630    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 20:57:41.438399 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.438246    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:41.438399 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:41.438360    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r2gd" podUID="278ac5cb-23c6-4748-b751-d5f3c53fdb23"
Sep 10 20:57:41.441294 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.440268    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 20:57:41.441294 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.440367    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 20:57:41.441294 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.440549    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 20:57:41.441294 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.440601    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vqdb7\""
Sep 10 20:57:41.441294 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.441234    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 20:57:41.441723 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.441433    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 20:57:41.441853 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.441825    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.442401 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.442338    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 20:57:41.442951 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.442932    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dnjhl"
Sep 10 20:57:41.445207 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.445187    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 20:57:41.445976 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.445193    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:57:41.445976 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.445476    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6pfzv\""
Sep 10 20:57:41.445976 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.445744    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ccgh9"
Sep 10 20:57:41.445976 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.445975    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:57:41.446242 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.446177    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 20:57:41.448826 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.448805    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 20:57:41.450046 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450026    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v6bl9\""
Sep 10 20:57:41.452427 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a82c7020-9c1d-41a8-97a5-a752e87fba5a-agent-certs\") pod \"konnectivity-agent-v6rlb\" (UID: \"a82c7020-9c1d-41a8-97a5-a752e87fba5a\") " pod="kube-system/konnectivity-agent-v6rlb"
Sep 10 20:57:41.452427 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-registration-dir\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.452427 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450433    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-kubelet\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.452427 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-var-lib-openvswitch\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.452427 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-cni-netd\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.452427 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.452427 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450574    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-sysconfig\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.452427 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a82c7020-9c1d-41a8-97a5-a752e87fba5a-konnectivity-ca\") pod \"konnectivity-agent-v6rlb\" (UID: \"a82c7020-9c1d-41a8-97a5-a752e87fba5a\") " pod="kube-system/konnectivity-agent-v6rlb"
Sep 10 20:57:41.452427 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pvzh\" (UniqueName: \"kubernetes.io/projected/278ac5cb-23c6-4748-b751-d5f3c53fdb23-kube-api-access-2pvzh\") pod \"network-metrics-daemon-4r2gd\" (UID: \"278ac5cb-23c6-4748-b751-d5f3c53fdb23\") " pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:41.452427 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.452427 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-sys-fs\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.452427 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-slash\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.452427 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-run-openvswitch\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.452427 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-run-ovn-kubernetes\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.452427 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v744t\" (UniqueName: \"kubernetes.io/projected/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-kube-api-access-v744t\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.452427 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450885    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddb8643c-235c-4694-b9dd-cd72d8986e42-hosts-file\") pod \"node-resolver-ps8fr\" (UID: \"ddb8643c-235c-4694-b9dd-cd72d8986e42\") " pod="openshift-dns/node-resolver-ps8fr"
Sep 10 20:57:41.453178 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-ovnkube-config\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.453178 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.450975    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-env-overrides\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.453178 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451015    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-ovn-node-metrics-cert\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.453178 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451047    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-modprobe-d\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.453178 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451085    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-kubernetes\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.453178 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-systemd\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.453178 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451155    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-cni-bin\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.453178 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-ovnkube-script-lib\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.453178 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-sysctl-conf\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.453178 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-run\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.453178 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-lib-modules\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.453178 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc55n\" (UniqueName: \"kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n\") pod \"network-check-target-lpwwj\" (UID: \"97ae132c-459a-4cba-bb9f-822279e84225\") " pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:57:41.453178 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-run-systemd\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.453178 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jdhr\" (UniqueName: \"kubernetes.io/projected/cf7c2283-19e2-435f-81bd-2645fd914c77-kube-api-access-5jdhr\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.453178 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-258t4\" (UniqueName: \"kubernetes.io/projected/ddb8643c-235c-4694-b9dd-cd72d8986e42-kube-api-access-258t4\") pod \"node-resolver-ps8fr\" (UID: \"ddb8643c-235c-4694-b9dd-cd72d8986e42\") " pod="openshift-dns/node-resolver-ps8fr"
Sep 10 20:57:41.453178 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs\") pod \"network-metrics-daemon-4r2gd\" (UID: \"278ac5cb-23c6-4748-b751-d5f3c53fdb23\") " pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:41.453926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-socket-dir\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.453926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-device-dir\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.453926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-sysctl-d\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.453926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-tuned\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.453926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddb8643c-235c-4694-b9dd-cd72d8986e42-tmp-dir\") pod \"node-resolver-ps8fr\" (UID: \"ddb8643c-235c-4694-b9dd-cd72d8986e42\") " pod="openshift-dns/node-resolver-ps8fr"
Sep 10 20:57:41.453926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl5b7\" (UniqueName: \"kubernetes.io/projected/f8007613-a73a-44e4-8507-e989280f250a-kube-api-access-bl5b7\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.453926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-etc-openvswitch\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.453926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-run-ovn\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.453926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451780    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-node-log\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.453926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-var-lib-kubelet\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.453926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-host\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.453926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf7c2283-19e2-435f-81bd-2645fd914c77-tmp\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.453926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-etc-selinux\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.453926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-systemd-units\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.453926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.451970    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-run-netns\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.453926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.452003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-log-socket\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.454763 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.452038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-sys\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.454763 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.452409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:41.454763 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:41.452610    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29b5p" podUID="8cefd1c8-a812-4cde-8fda-cc35af6e9833"
Sep 10 20:57:41.454763 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.452861    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.455364 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.455342    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.455964 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.455939    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7xkcp\""
Sep 10 20:57:41.457805 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.457781    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 20:57:41.458108 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.458089    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 20:57:41.458324 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.458307    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 20:57:41.459423 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.459402    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vv2nt\""
Sep 10 20:57:41.459667 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.459649    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 20:57:41.459909 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.459891    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 20:57:41.460141 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.460123    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 20:57:41.460352 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.460335    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 20:57:41.462297 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.461339    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 20:57:41.462297 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.461571    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 20:57:41.462297 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.461899    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7674z\""
Sep 10 20:57:41.462297 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.462194    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 20:57:41.527398 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.527369    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6sqlh"
Sep 10 20:57:41.539097 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.539032    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6sqlh"
Sep 10 20:57:41.539818 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.539795    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 20:57:41.552440 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.552411    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-run-ovn-kubernetes\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.552554 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.552461    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.552554 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.552492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddb8643c-235c-4694-b9dd-cd72d8986e42-hosts-file\") pod \"node-resolver-ps8fr\" (UID: \"ddb8643c-235c-4694-b9dd-cd72d8986e42\") " pod="openshift-dns/node-resolver-ps8fr"
Sep 10 20:57:41.552554 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.552544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-ovnkube-config\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.552728 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.552546    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-run-ovn-kubernetes\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.552728 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.552564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddb8643c-235c-4694-b9dd-cd72d8986e42-hosts-file\") pod \"node-resolver-ps8fr\" (UID: \"ddb8643c-235c-4694-b9dd-cd72d8986e42\") " pod="openshift-dns/node-resolver-ps8fr"
Sep 10 20:57:41.552728 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.552600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-modprobe-d\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.552728 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.552625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-run\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.552932 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.552813    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-modprobe-d\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.552932 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.552837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-lib-modules\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.552932 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.552866    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-multus-cni-dir\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.553057 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553004    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-lib-modules\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.553057 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59e8332d-e77e-4229-82a5-e3cf8ac84d81-cni-binary-copy\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.553137 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553059    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-cni-bin\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.553137 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553083    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-ovnkube-config\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.553207 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553143    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-cni-bin\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.553207 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-run\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.553303 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-ovnkube-script-lib\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.553303 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-sysctl-conf\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.553303 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b97jb\" (UniqueName: \"kubernetes.io/projected/38d7a53b-34ee-421c-9e3e-3fafd56dcc09-kube-api-access-b97jb\") pod \"iptables-alerter-dnjhl\" (UID: \"38d7a53b-34ee-421c-9e3e-3fafd56dcc09\") " pod="openshift-network-operator/iptables-alerter-dnjhl"
Sep 10 20:57:41.553445 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27105194-1643-4b6a-ae7b-2e0d76967bb6-serviceca\") pod \"node-ca-ccgh9\" (UID: \"27105194-1643-4b6a-ae7b-2e0d76967bb6\") " pod="openshift-image-registry/node-ca-ccgh9"
Sep 10 20:57:41.553445 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553325    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-var-lib-cni-bin\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.553445 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/59e8332d-e77e-4229-82a5-e3cf8ac84d81-multus-daemon-config\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.553445 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-run-multus-certs\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.553584 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc55n\" (UniqueName: \"kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n\") pod \"network-check-target-lpwwj\" (UID: \"97ae132c-459a-4cba-bb9f-822279e84225\") " pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:57:41.553584 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-run-systemd\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.553584 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-sysctl-conf\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.553584 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553527    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38d7a53b-34ee-421c-9e3e-3fafd56dcc09-iptables-alerter-script\") pod \"iptables-alerter-dnjhl\" (UID: \"38d7a53b-34ee-421c-9e3e-3fafd56dcc09\") " pod="openshift-network-operator/iptables-alerter-dnjhl"
Sep 10 20:57:41.553584 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-var-lib-kubelet\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.553584 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-etc-kubernetes\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.553882 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.553882 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs\") pod \"network-metrics-daemon-4r2gd\" (UID: \"278ac5cb-23c6-4748-b751-d5f3c53fdb23\") " pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:41.553882 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-socket-dir\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.553882 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553698    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8cefd1c8-a812-4cde-8fda-cc35af6e9833-kubelet-config\") pod \"global-pull-secret-syncer-29b5p\" (UID: \"8cefd1c8-a812-4cde-8fda-cc35af6e9833\") " pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:41.553882 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret\") pod \"global-pull-secret-syncer-29b5p\" (UID: \"8cefd1c8-a812-4cde-8fda-cc35af6e9833\") " pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:41.553882 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-cnibin\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.553882 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553781    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-cni-binary-copy\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.553882 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddb8643c-235c-4694-b9dd-cd72d8986e42-tmp-dir\") pod \"node-resolver-ps8fr\" (UID: \"ddb8643c-235c-4694-b9dd-cd72d8986e42\") " pod="openshift-dns/node-resolver-ps8fr"
Sep 10 20:57:41.553882 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553814    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-ovnkube-script-lib\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.553882 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-node-log\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.553882 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf7c2283-19e2-435f-81bd-2645fd914c77-tmp\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.554488 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sxzq\" (UniqueName: \"kubernetes.io/projected/59e8332d-e77e-4229-82a5-e3cf8ac84d81-kube-api-access-6sxzq\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.554488 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-etc-selinux\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.554488 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-systemd-units\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.554488 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.553984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-sys\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.554488 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-system-cni-dir\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.554488 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-system-cni-dir\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.554488 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.554488 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wbhc\" (UniqueName: \"kubernetes.io/projected/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-kube-api-access-2wbhc\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.554488 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-node-log\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.554488 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a82c7020-9c1d-41a8-97a5-a752e87fba5a-agent-certs\") pod \"konnectivity-agent-v6rlb\" (UID: \"a82c7020-9c1d-41a8-97a5-a752e87fba5a\") " pod="kube-system/konnectivity-agent-v6rlb"
Sep 10 20:57:41.554488 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554191    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-run-systemd\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.554488 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:41.554202    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:41.554488 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-kubelet\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.554488 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:41.554313    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs podName:278ac5cb-23c6-4748-b751-d5f3c53fdb23 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:42.054265393 +0000 UTC m=+3.029925436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs") pod "network-metrics-daemon-4r2gd" (UID: "278ac5cb-23c6-4748-b751-d5f3c53fdb23") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:41.554488 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554316    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-socket-dir\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.554488 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-kubelet\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-cni-netd\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554596    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-sysconfig\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27105194-1643-4b6a-ae7b-2e0d76967bb6-host\") pod \"node-ca-ccgh9\" (UID: \"27105194-1643-4b6a-ae7b-2e0d76967bb6\") " pod="openshift-image-registry/node-ca-ccgh9"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554665    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-sys\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a82c7020-9c1d-41a8-97a5-a752e87fba5a-konnectivity-ca\") pod \"konnectivity-agent-v6rlb\" (UID: \"a82c7020-9c1d-41a8-97a5-a752e87fba5a\") " pod="kube-system/konnectivity-agent-v6rlb"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pvzh\" (UniqueName: \"kubernetes.io/projected/278ac5cb-23c6-4748-b751-d5f3c53fdb23-kube-api-access-2pvzh\") pod \"network-metrics-daemon-4r2gd\" (UID: \"278ac5cb-23c6-4748-b751-d5f3c53fdb23\") " pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddb8643c-235c-4694-b9dd-cd72d8986e42-tmp-dir\") pod \"node-resolver-ps8fr\" (UID: \"ddb8643c-235c-4694-b9dd-cd72d8986e42\") " pod="openshift-dns/node-resolver-ps8fr"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554915    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v744t\" (UniqueName: \"kubernetes.io/projected/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-kube-api-access-v744t\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-kubernetes\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.554985    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-etc-selinux\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-systemd\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555026    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-sysconfig\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-var-lib-cni-multus\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-env-overrides\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555076    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-systemd-units\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555053    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 20:57:41.555301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-ovn-node-metrics-cert\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.556181 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-hostroot\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.556181 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65wxm\" (UniqueName: \"kubernetes.io/projected/27105194-1643-4b6a-ae7b-2e0d76967bb6-kube-api-access-65wxm\") pod \"node-ca-ccgh9\" (UID: \"27105194-1643-4b6a-ae7b-2e0d76967bb6\") " pod="openshift-image-registry/node-ca-ccgh9"
Sep 10 20:57:41.556181 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdhr\" (UniqueName: \"kubernetes.io/projected/cf7c2283-19e2-435f-81bd-2645fd914c77-kube-api-access-5jdhr\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.556181 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8cefd1c8-a812-4cde-8fda-cc35af6e9833-dbus\") pod \"global-pull-secret-syncer-29b5p\" (UID: \"8cefd1c8-a812-4cde-8fda-cc35af6e9833\") " pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:41.556181 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555417    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a82c7020-9c1d-41a8-97a5-a752e87fba5a-konnectivity-ca\") pod \"konnectivity-agent-v6rlb\" (UID: \"a82c7020-9c1d-41a8-97a5-a752e87fba5a\") " pod="kube-system/konnectivity-agent-v6rlb"
Sep 10 20:57:41.556181 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555427    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.556181 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-258t4\" (UniqueName: \"kubernetes.io/projected/ddb8643c-235c-4694-b9dd-cd72d8986e42-kube-api-access-258t4\") pod \"node-resolver-ps8fr\" (UID: \"ddb8643c-235c-4694-b9dd-cd72d8986e42\") " pod="openshift-dns/node-resolver-ps8fr"
Sep 10 20:57:41.556181 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-device-dir\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.556181 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555484    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-cni-netd\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.556181 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-kubernetes\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.556181 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-sysctl-d\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.556181 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555558    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-tuned\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.556181 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-cnibin\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.556181 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555662    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-multus-socket-dir-parent\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.556181 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl5b7\" (UniqueName: \"kubernetes.io/projected/f8007613-a73a-44e4-8507-e989280f250a-kube-api-access-bl5b7\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.556181 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-etc-openvswitch\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.556181 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555782    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-device-dir\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-run-ovn\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555834    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-systemd\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-var-lib-kubelet\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555845    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-env-overrides\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-host\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555906    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-var-lib-kubelet\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-run-netns\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555986    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-run-ovn\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.555986    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-host\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556028    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-log-socket\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-run-netns\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556070    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-sysctl-d\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-etc-openvswitch\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556130    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-log-socket\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556187    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38d7a53b-34ee-421c-9e3e-3fafd56dcc09-host-slash\") pod \"iptables-alerter-dnjhl\" (UID: \"38d7a53b-34ee-421c-9e3e-3fafd56dcc09\") " pod="openshift-network-operator/iptables-alerter-dnjhl"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-run-netns\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-registration-dir\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.557072 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-var-lib-openvswitch\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.557972 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-registration-dir\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.557972 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-var-lib-openvswitch\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.557972 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.557972 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-run-k8s-cni-cncf-io\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.557972 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556474    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-multus-conf-dir\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.557972 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556493    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-sys-fs\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.557972 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-slash\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.557972 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556515    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.557972 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-run-openvswitch\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.557972 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556603    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-run-openvswitch\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.557972 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-host-slash\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.557972 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.556656    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8007613-a73a-44e4-8507-e989280f250a-sys-fs\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.559725 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.559692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a82c7020-9c1d-41a8-97a5-a752e87fba5a-agent-certs\") pod \"konnectivity-agent-v6rlb\" (UID: \"a82c7020-9c1d-41a8-97a5-a752e87fba5a\") " pod="kube-system/konnectivity-agent-v6rlb"
Sep 10 20:57:41.559867 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.559716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cf7c2283-19e2-435f-81bd-2645fd914c77-etc-tuned\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.559867 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.559793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf7c2283-19e2-435f-81bd-2645fd914c77-tmp\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.560110 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.560088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-ovn-node-metrics-cert\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.563496 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:41.563264    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:57:41.563496 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:41.563305    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:57:41.563496 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:41.563319    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qc55n for pod openshift-network-diagnostics/network-check-target-lpwwj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:41.563496 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:41.563389    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n podName:97ae132c-459a-4cba-bb9f-822279e84225 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:42.063370682 +0000 UTC m=+3.039030715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qc55n" (UniqueName: "kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n") pod "network-check-target-lpwwj" (UID: "97ae132c-459a-4cba-bb9f-822279e84225") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:41.566611 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.566587    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v744t\" (UniqueName: \"kubernetes.io/projected/ab02629e-1b6c-4937-bf8e-26f31f42ed5b-kube-api-access-v744t\") pod \"ovnkube-node-m777h\" (UID: \"ab02629e-1b6c-4937-bf8e-26f31f42ed5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.568433 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.568409    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jdhr\" (UniqueName: \"kubernetes.io/projected/cf7c2283-19e2-435f-81bd-2645fd914c77-kube-api-access-5jdhr\") pod \"tuned-qp845\" (UID: \"cf7c2283-19e2-435f-81bd-2645fd914c77\") " pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.568537 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.568418    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pvzh\" (UniqueName: \"kubernetes.io/projected/278ac5cb-23c6-4748-b751-d5f3c53fdb23-kube-api-access-2pvzh\") pod \"network-metrics-daemon-4r2gd\" (UID: \"278ac5cb-23c6-4748-b751-d5f3c53fdb23\") " pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:41.569627 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.569600    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-258t4\" (UniqueName: \"kubernetes.io/projected/ddb8643c-235c-4694-b9dd-cd72d8986e42-kube-api-access-258t4\") pod \"node-resolver-ps8fr\" (UID: \"ddb8643c-235c-4694-b9dd-cd72d8986e42\") " pod="openshift-dns/node-resolver-ps8fr"
Sep 10 20:57:41.569724 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.569691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl5b7\" (UniqueName: \"kubernetes.io/projected/f8007613-a73a-44e4-8507-e989280f250a-kube-api-access-bl5b7\") pod \"aws-ebs-csi-driver-node-nhq87\" (UID: \"f8007613-a73a-44e4-8507-e989280f250a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.572863 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.572821    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-6.ec2.internal" event={"ID":"fb31595c699a42d1829717a278752f0e","Type":"ContainerStarted","Data":"f744090471859bc84dc5e641fd2a88cdf27fed86f569c28d44f172fb3670c127"}
Sep 10 20:57:41.574290 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.574265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal" event={"ID":"fdce11d1461a1b096b934b218406a2b8","Type":"ContainerStarted","Data":"0ecfa2de739713e4a1303c7a55246901bcd3aa08605966f16843d8a6eb919f3d"}
Sep 10 20:57:41.657703 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.657671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-etc-kubernetes\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.657856 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.657713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.657856 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.657804    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-etc-kubernetes\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.657982 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.657871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8cefd1c8-a812-4cde-8fda-cc35af6e9833-kubelet-config\") pod \"global-pull-secret-syncer-29b5p\" (UID: \"8cefd1c8-a812-4cde-8fda-cc35af6e9833\") " pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:41.657982 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.657900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret\") pod \"global-pull-secret-syncer-29b5p\" (UID: \"8cefd1c8-a812-4cde-8fda-cc35af6e9833\") " pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:41.657982 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.657927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-cnibin\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.658133 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-cni-binary-copy\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.658133 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658027    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8cefd1c8-a812-4cde-8fda-cc35af6e9833-kubelet-config\") pod \"global-pull-secret-syncer-29b5p\" (UID: \"8cefd1c8-a812-4cde-8fda-cc35af6e9833\") " pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:41.658133 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sxzq\" (UniqueName: \"kubernetes.io/projected/59e8332d-e77e-4229-82a5-e3cf8ac84d81-kube-api-access-6sxzq\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.658133 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:41.658098    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:41.658367 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658135    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-system-cni-dir\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.658367 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:41.658165    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret podName:8cefd1c8-a812-4cde-8fda-cc35af6e9833 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:42.158145717 +0000 UTC m=+3.133805752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret") pod "global-pull-secret-syncer-29b5p" (UID: "8cefd1c8-a812-4cde-8fda-cc35af6e9833") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:41.658367 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-system-cni-dir\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.658367 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-system-cni-dir\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.658367 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.658367 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658291    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wbhc\" (UniqueName: \"kubernetes.io/projected/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-kube-api-access-2wbhc\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.658367 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27105194-1643-4b6a-ae7b-2e0d76967bb6-host\") pod \"node-ca-ccgh9\" (UID: \"27105194-1643-4b6a-ae7b-2e0d76967bb6\") " pod="openshift-image-registry/node-ca-ccgh9"
Sep 10 20:57:41.658367 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-var-lib-cni-multus\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658427    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-cnibin\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658432    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-hostroot\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658466    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65wxm\" (UniqueName: \"kubernetes.io/projected/27105194-1643-4b6a-ae7b-2e0d76967bb6-kube-api-access-65wxm\") pod \"node-ca-ccgh9\" (UID: \"27105194-1643-4b6a-ae7b-2e0d76967bb6\") " pod="openshift-image-registry/node-ca-ccgh9"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658479    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-hostroot\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8cefd1c8-a812-4cde-8fda-cc35af6e9833-dbus\") pod \"global-pull-secret-syncer-29b5p\" (UID: \"8cefd1c8-a812-4cde-8fda-cc35af6e9833\") " pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658397    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-var-lib-cni-multus\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-cnibin\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658549    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-system-cni-dir\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-cni-binary-copy\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-multus-socket-dir-parent\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-cnibin\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38d7a53b-34ee-421c-9e3e-3fafd56dcc09-host-slash\") pod \"iptables-alerter-dnjhl\" (UID: \"38d7a53b-34ee-421c-9e3e-3fafd56dcc09\") " pod="openshift-network-operator/iptables-alerter-dnjhl"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658711    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-multus-socket-dir-parent\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658736    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8cefd1c8-a812-4cde-8fda-cc35af6e9833-dbus\") pod \"global-pull-secret-syncer-29b5p\" (UID: \"8cefd1c8-a812-4cde-8fda-cc35af6e9833\") " pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658751    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-run-netns\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38d7a53b-34ee-421c-9e3e-3fafd56dcc09-host-slash\") pod \"iptables-alerter-dnjhl\" (UID: \"38d7a53b-34ee-421c-9e3e-3fafd56dcc09\") " pod="openshift-network-operator/iptables-alerter-dnjhl"
Sep 10 20:57:41.658774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-run-k8s-cni-cncf-io\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658797    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27105194-1643-4b6a-ae7b-2e0d76967bb6-host\") pod \"node-ca-ccgh9\" (UID: \"27105194-1643-4b6a-ae7b-2e0d76967bb6\") " pod="openshift-image-registry/node-ca-ccgh9"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-multus-conf-dir\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-run-k8s-cni-cncf-io\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658847    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-multus-conf-dir\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658836    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-run-netns\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-multus-cni-dir\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59e8332d-e77e-4229-82a5-e3cf8ac84d81-cni-binary-copy\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b97jb\" (UniqueName: \"kubernetes.io/projected/38d7a53b-34ee-421c-9e3e-3fafd56dcc09-kube-api-access-b97jb\") pod \"iptables-alerter-dnjhl\" (UID: \"38d7a53b-34ee-421c-9e3e-3fafd56dcc09\") " pod="openshift-network-operator/iptables-alerter-dnjhl"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27105194-1643-4b6a-ae7b-2e0d76967bb6-serviceca\") pod \"node-ca-ccgh9\" (UID: \"27105194-1643-4b6a-ae7b-2e0d76967bb6\") " pod="openshift-image-registry/node-ca-ccgh9"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658967    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-multus-cni-dir\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.658971    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-var-lib-cni-bin\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.659056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/59e8332d-e77e-4229-82a5-e3cf8ac84d81-multus-daemon-config\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.659084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-run-multus-certs\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.659138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38d7a53b-34ee-421c-9e3e-3fafd56dcc09-iptables-alerter-script\") pod \"iptables-alerter-dnjhl\" (UID: \"38d7a53b-34ee-421c-9e3e-3fafd56dcc09\") " pod="openshift-network-operator/iptables-alerter-dnjhl"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.659166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-var-lib-kubelet\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.659684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.659174    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.660295 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.659234    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-var-lib-kubelet\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.660295 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.659264    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-run-multus-certs\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.660295 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.659333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/59e8332d-e77e-4229-82a5-e3cf8ac84d81-host-var-lib-cni-bin\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.660295 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.659443    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27105194-1643-4b6a-ae7b-2e0d76967bb6-serviceca\") pod \"node-ca-ccgh9\" (UID: \"27105194-1643-4b6a-ae7b-2e0d76967bb6\") " pod="openshift-image-registry/node-ca-ccgh9"
Sep 10 20:57:41.660295 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.659548    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59e8332d-e77e-4229-82a5-e3cf8ac84d81-cni-binary-copy\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.660295 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.659781    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/59e8332d-e77e-4229-82a5-e3cf8ac84d81-multus-daemon-config\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.660295 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.659850    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38d7a53b-34ee-421c-9e3e-3fafd56dcc09-iptables-alerter-script\") pod \"iptables-alerter-dnjhl\" (UID: \"38d7a53b-34ee-421c-9e3e-3fafd56dcc09\") " pod="openshift-network-operator/iptables-alerter-dnjhl"
Sep 10 20:57:41.669821 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.669766    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wbhc\" (UniqueName: \"kubernetes.io/projected/1ee23d1e-61de-48f1-85c6-6d1ef3437ba2-kube-api-access-2wbhc\") pod \"multus-additional-cni-plugins-qg95l\" (UID: \"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2\") " pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:41.670654 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.670627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b97jb\" (UniqueName: \"kubernetes.io/projected/38d7a53b-34ee-421c-9e3e-3fafd56dcc09-kube-api-access-b97jb\") pod \"iptables-alerter-dnjhl\" (UID: \"38d7a53b-34ee-421c-9e3e-3fafd56dcc09\") " pod="openshift-network-operator/iptables-alerter-dnjhl"
Sep 10 20:57:41.670993 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.670973    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sxzq\" (UniqueName: \"kubernetes.io/projected/59e8332d-e77e-4229-82a5-e3cf8ac84d81-kube-api-access-6sxzq\") pod \"multus-j5bg8\" (UID: \"59e8332d-e77e-4229-82a5-e3cf8ac84d81\") " pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.671421 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.671399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65wxm\" (UniqueName: \"kubernetes.io/projected/27105194-1643-4b6a-ae7b-2e0d76967bb6-kube-api-access-65wxm\") pod \"node-ca-ccgh9\" (UID: \"27105194-1643-4b6a-ae7b-2e0d76967bb6\") " pod="openshift-image-registry/node-ca-ccgh9"
Sep 10 20:57:41.742488 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.742458    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v6rlb"
Sep 10 20:57:41.753779 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.753619    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ps8fr"
Sep 10 20:57:41.764875 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.764852    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87"
Sep 10 20:57:41.771372 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.771352    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:57:41.779373 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.779355    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qp845"
Sep 10 20:57:41.790300 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.790278    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dnjhl"
Sep 10 20:57:41.804357 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.804338    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ccgh9"
Sep 10 20:57:41.813253 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.813232    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j5bg8"
Sep 10 20:57:41.818982 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:41.818961    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qg95l"
Sep 10 20:57:42.062488 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:42.062421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs\") pod \"network-metrics-daemon-4r2gd\" (UID: \"278ac5cb-23c6-4748-b751-d5f3c53fdb23\") " pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:42.062645 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:42.062619    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:42.062728 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:42.062714    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs podName:278ac5cb-23c6-4748-b751-d5f3c53fdb23 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:43.062688516 +0000 UTC m=+4.038348559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs") pod "network-metrics-daemon-4r2gd" (UID: "278ac5cb-23c6-4748-b751-d5f3c53fdb23") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:42.163454 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:42.163417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc55n\" (UniqueName: \"kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n\") pod \"network-check-target-lpwwj\" (UID: \"97ae132c-459a-4cba-bb9f-822279e84225\") " pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:57:42.163645 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:42.163479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret\") pod \"global-pull-secret-syncer-29b5p\" (UID: \"8cefd1c8-a812-4cde-8fda-cc35af6e9833\") " pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:42.163645 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:42.163605    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:42.163645 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:42.163620    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:57:42.163645 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:42.163641    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:57:42.163871 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:42.163655    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qc55n for pod openshift-network-diagnostics/network-check-target-lpwwj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:42.163871 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:42.163670    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret podName:8cefd1c8-a812-4cde-8fda-cc35af6e9833 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:43.163651624 +0000 UTC m=+4.139311652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret") pod "global-pull-secret-syncer-29b5p" (UID: "8cefd1c8-a812-4cde-8fda-cc35af6e9833") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:42.163871 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:42.163706    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n podName:97ae132c-459a-4cba-bb9f-822279e84225 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:43.163688616 +0000 UTC m=+4.139348647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qc55n" (UniqueName: "kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n") pod "network-check-target-lpwwj" (UID: "97ae132c-459a-4cba-bb9f-822279e84225") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:42.542805 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:42.542717    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:52:42 +0000 UTC" deadline="2028-03-14 09:22:36.489998945 +0000 UTC"
Sep 10 20:57:42.542805 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:42.542752    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13212h24m53.947250495s"
Sep 10 20:57:42.792200 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:42.792169    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab02629e_1b6c_4937_bf8e_26f31f42ed5b.slice/crio-52517d1c97aef9b504e104ac44d37bf5959aa69e4ea563360e03ae5eae387264 WatchSource:0}: Error finding container 52517d1c97aef9b504e104ac44d37bf5959aa69e4ea563360e03ae5eae387264: Status 404 returned error can't find the container with id 52517d1c97aef9b504e104ac44d37bf5959aa69e4ea563360e03ae5eae387264
Sep 10 20:57:42.793252 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:42.793193    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7c2283_19e2_435f_81bd_2645fd914c77.slice/crio-b1598aea83c3b3839a4f328a74bbc0a2b1ce985de63f42f94caa364d334d6db1 WatchSource:0}: Error finding container b1598aea83c3b3839a4f328a74bbc0a2b1ce985de63f42f94caa364d334d6db1: Status 404 returned error can't find the container with id b1598aea83c3b3839a4f328a74bbc0a2b1ce985de63f42f94caa364d334d6db1
Sep 10 20:57:42.794541 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:42.794493    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27105194_1643_4b6a_ae7b_2e0d76967bb6.slice/crio-ff5d43ae12e1e3aa0ac0de7005afe502ebd53686605a43abf942147f2e59f385 WatchSource:0}: Error finding container ff5d43ae12e1e3aa0ac0de7005afe502ebd53686605a43abf942147f2e59f385: Status 404 returned error can't find the container with id ff5d43ae12e1e3aa0ac0de7005afe502ebd53686605a43abf942147f2e59f385
Sep 10 20:57:42.798298 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:42.797888    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb8643c_235c_4694_b9dd_cd72d8986e42.slice/crio-d93374b9bec962f642aa5e7b44164057425f80b45abc996927e52f7fe089459c WatchSource:0}: Error finding container d93374b9bec962f642aa5e7b44164057425f80b45abc996927e52f7fe089459c: Status 404 returned error can't find the container with id d93374b9bec962f642aa5e7b44164057425f80b45abc996927e52f7fe089459c
Sep 10 20:57:42.800486 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:42.800454    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82c7020_9c1d_41a8_97a5_a752e87fba5a.slice/crio-6028c830b7d1ccb4460f8dd1985b30361b841447521141dc2d1f401205eb999b WatchSource:0}: Error finding container 6028c830b7d1ccb4460f8dd1985b30361b841447521141dc2d1f401205eb999b: Status 404 returned error can't find the container with id 6028c830b7d1ccb4460f8dd1985b30361b841447521141dc2d1f401205eb999b
Sep 10 20:57:42.802291 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:42.802264    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee23d1e_61de_48f1_85c6_6d1ef3437ba2.slice/crio-caebe1dc80f9c7ec7f35508482da6a13515ef192080e2185043f2ec05d901bed WatchSource:0}: Error finding container caebe1dc80f9c7ec7f35508482da6a13515ef192080e2185043f2ec05d901bed: Status 404 returned error can't find the container with id caebe1dc80f9c7ec7f35508482da6a13515ef192080e2185043f2ec05d901bed
Sep 10 20:57:42.803381 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:42.803178    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8007613_a73a_44e4_8507_e989280f250a.slice/crio-dfadf63de22d2fb7088cc69c4f24bf743932dc2a06254965e9bc95cfc4c6b8d7 WatchSource:0}: Error finding container dfadf63de22d2fb7088cc69c4f24bf743932dc2a06254965e9bc95cfc4c6b8d7: Status 404 returned error can't find the container with id dfadf63de22d2fb7088cc69c4f24bf743932dc2a06254965e9bc95cfc4c6b8d7
Sep 10 20:57:42.804099 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:42.804018    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d7a53b_34ee_421c_9e3e_3fafd56dcc09.slice/crio-e2e3cb0c393aee99909cc073083e83f40c9a623927fac51ed17c729f191e5e4d WatchSource:0}: Error finding container e2e3cb0c393aee99909cc073083e83f40c9a623927fac51ed17c729f191e5e4d: Status 404 returned error can't find the container with id e2e3cb0c393aee99909cc073083e83f40c9a623927fac51ed17c729f191e5e4d
Sep 10 20:57:42.805869 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:57:42.805835    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e8332d_e77e_4229_82a5_e3cf8ac84d81.slice/crio-cc0e1f6ccfe4b8879e23e3d515b02e0d76e88e611e443a1939f017dd71d761bb WatchSource:0}: Error finding container cc0e1f6ccfe4b8879e23e3d515b02e0d76e88e611e443a1939f017dd71d761bb: Status 404 returned error can't find the container with id cc0e1f6ccfe4b8879e23e3d515b02e0d76e88e611e443a1939f017dd71d761bb
Sep 10 20:57:43.069614 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.069586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs\") pod \"network-metrics-daemon-4r2gd\" (UID: \"278ac5cb-23c6-4748-b751-d5f3c53fdb23\") " pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:43.069743 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:43.069727    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:43.069793 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:43.069783    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs podName:278ac5cb-23c6-4748-b751-d5f3c53fdb23 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:45.069769284 +0000 UTC m=+6.045429315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs") pod "network-metrics-daemon-4r2gd" (UID: "278ac5cb-23c6-4748-b751-d5f3c53fdb23") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:43.170814 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.170783    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc55n\" (UniqueName: \"kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n\") pod \"network-check-target-lpwwj\" (UID: \"97ae132c-459a-4cba-bb9f-822279e84225\") " pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:57:43.170938 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.170828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret\") pod \"global-pull-secret-syncer-29b5p\" (UID: \"8cefd1c8-a812-4cde-8fda-cc35af6e9833\") " pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:43.170977 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:43.170939    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:43.170977 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:43.170955    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:57:43.170977 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:43.170972    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:57:43.171067 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:43.170985    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qc55n for pod openshift-network-diagnostics/network-check-target-lpwwj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:43.171067 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:43.171005    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret podName:8cefd1c8-a812-4cde-8fda-cc35af6e9833 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:45.170987226 +0000 UTC m=+6.146647271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret") pod "global-pull-secret-syncer-29b5p" (UID: "8cefd1c8-a812-4cde-8fda-cc35af6e9833") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:43.171067 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:43.171032    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n podName:97ae132c-459a-4cba-bb9f-822279e84225 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:45.171021049 +0000 UTC m=+6.146681091 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qc55n" (UniqueName: "kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n") pod "network-check-target-lpwwj" (UID: "97ae132c-459a-4cba-bb9f-822279e84225") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:43.359898 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.359627    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:57:43.543835 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.542888    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:52:42 +0000 UTC" deadline="2028-04-19 07:57:44.451277114 +0000 UTC"
Sep 10 20:57:43.543835 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.542931    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14075h0m0.908350655s"
Sep 10 20:57:43.574981 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.574949    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:43.575198 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:43.575169    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r2gd" podUID="278ac5cb-23c6-4748-b751-d5f3c53fdb23"
Sep 10 20:57:43.575413 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.575394    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:43.575570 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:43.575546    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29b5p" podUID="8cefd1c8-a812-4cde-8fda-cc35af6e9833"
Sep 10 20:57:43.575648 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.575631    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:57:43.575759 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:43.575739    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpwwj" podUID="97ae132c-459a-4cba-bb9f-822279e84225"
Sep 10 20:57:43.594130 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.594098    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qg95l" event={"ID":"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2","Type":"ContainerStarted","Data":"caebe1dc80f9c7ec7f35508482da6a13515ef192080e2185043f2ec05d901bed"}
Sep 10 20:57:43.611430 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.611353    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j5bg8" event={"ID":"59e8332d-e77e-4229-82a5-e3cf8ac84d81","Type":"ContainerStarted","Data":"cc0e1f6ccfe4b8879e23e3d515b02e0d76e88e611e443a1939f017dd71d761bb"}
Sep 10 20:57:43.617492 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.617452    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v6rlb" event={"ID":"a82c7020-9c1d-41a8-97a5-a752e87fba5a","Type":"ContainerStarted","Data":"6028c830b7d1ccb4460f8dd1985b30361b841447521141dc2d1f401205eb999b"}
Sep 10 20:57:43.623781 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.623733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ps8fr" event={"ID":"ddb8643c-235c-4694-b9dd-cd72d8986e42","Type":"ContainerStarted","Data":"d93374b9bec962f642aa5e7b44164057425f80b45abc996927e52f7fe089459c"}
Sep 10 20:57:43.639758 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.639724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ccgh9" event={"ID":"27105194-1643-4b6a-ae7b-2e0d76967bb6","Type":"ContainerStarted","Data":"ff5d43ae12e1e3aa0ac0de7005afe502ebd53686605a43abf942147f2e59f385"}
Sep 10 20:57:43.648901 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.648864    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qp845" event={"ID":"cf7c2283-19e2-435f-81bd-2645fd914c77","Type":"ContainerStarted","Data":"b1598aea83c3b3839a4f328a74bbc0a2b1ce985de63f42f94caa364d334d6db1"}
Sep 10 20:57:43.666709 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.666675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m777h" event={"ID":"ab02629e-1b6c-4937-bf8e-26f31f42ed5b","Type":"ContainerStarted","Data":"52517d1c97aef9b504e104ac44d37bf5959aa69e4ea563360e03ae5eae387264"}
Sep 10 20:57:43.675056 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.675016    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-6.ec2.internal" event={"ID":"fb31595c699a42d1829717a278752f0e","Type":"ContainerStarted","Data":"ac9c70918cab0f05e52a0f9d168947911f5a8abcd1577870281b54daa3e14846"}
Sep 10 20:57:43.686766 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.686739    2581 generic.go:356] "Generic (PLEG): container finished" podID="fdce11d1461a1b096b934b218406a2b8" containerID="6b1db660710b965eb986f81d7cd8a9bc16838ca19f64da5763cd79b1b94d00c2" exitCode=0
Sep 10 20:57:43.686868 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.686820    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal" event={"ID":"fdce11d1461a1b096b934b218406a2b8","Type":"ContainerDied","Data":"6b1db660710b965eb986f81d7cd8a9bc16838ca19f64da5763cd79b1b94d00c2"}
Sep 10 20:57:43.700264 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.700058    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dnjhl" event={"ID":"38d7a53b-34ee-421c-9e3e-3fafd56dcc09","Type":"ContainerStarted","Data":"e2e3cb0c393aee99909cc073083e83f40c9a623927fac51ed17c729f191e5e4d"}
Sep 10 20:57:43.703484 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.703223    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87" event={"ID":"f8007613-a73a-44e4-8507-e989280f250a","Type":"ContainerStarted","Data":"dfadf63de22d2fb7088cc69c4f24bf743932dc2a06254965e9bc95cfc4c6b8d7"}
Sep 10 20:57:43.720844 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:43.719577    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-6.ec2.internal" podStartSLOduration=2.719493855 podStartE2EDuration="2.719493855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:43.700039806 +0000 UTC m=+4.675699858" watchObservedRunningTime="2026-09-10 20:57:43.719493855 +0000 UTC m=+4.695153906"
Sep 10 20:57:44.722607 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:44.722574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal" event={"ID":"fdce11d1461a1b096b934b218406a2b8","Type":"ContainerStarted","Data":"0ecf1eee0b14e55480320b1b96a35cedba3a57003eb2dcb9fadddafc8eae9e37"}
Sep 10 20:57:45.090154 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:45.089547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs\") pod \"network-metrics-daemon-4r2gd\" (UID: \"278ac5cb-23c6-4748-b751-d5f3c53fdb23\") " pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:45.090154 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:45.089742    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:45.090154 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:45.089807    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs podName:278ac5cb-23c6-4748-b751-d5f3c53fdb23 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:49.089787162 +0000 UTC m=+10.065447197 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs") pod "network-metrics-daemon-4r2gd" (UID: "278ac5cb-23c6-4748-b751-d5f3c53fdb23") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:45.190919 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:45.190878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc55n\" (UniqueName: \"kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n\") pod \"network-check-target-lpwwj\" (UID: \"97ae132c-459a-4cba-bb9f-822279e84225\") " pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:57:45.191202 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:45.190947    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret\") pod \"global-pull-secret-syncer-29b5p\" (UID: \"8cefd1c8-a812-4cde-8fda-cc35af6e9833\") " pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:45.191202 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:45.191075    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:45.191202 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:45.191139    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret podName:8cefd1c8-a812-4cde-8fda-cc35af6e9833 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:49.191120428 +0000 UTC m=+10.166780480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret") pod "global-pull-secret-syncer-29b5p" (UID: "8cefd1c8-a812-4cde-8fda-cc35af6e9833") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:45.191613 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:45.191589    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:57:45.191613 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:45.191616    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:57:45.191779 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:45.191629    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qc55n for pod openshift-network-diagnostics/network-check-target-lpwwj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:45.191779 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:45.191682    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n podName:97ae132c-459a-4cba-bb9f-822279e84225 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:49.191663834 +0000 UTC m=+10.167323884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qc55n" (UniqueName: "kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n") pod "network-check-target-lpwwj" (UID: "97ae132c-459a-4cba-bb9f-822279e84225") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:45.572026 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:45.570942    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:45.572026 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:45.571150    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29b5p" podUID="8cefd1c8-a812-4cde-8fda-cc35af6e9833"
Sep 10 20:57:45.572026 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:45.571565    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:57:45.572026 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:45.571662    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpwwj" podUID="97ae132c-459a-4cba-bb9f-822279e84225"
Sep 10 20:57:45.572026 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:45.571713    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:45.572026 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:45.571798    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r2gd" podUID="278ac5cb-23c6-4748-b751-d5f3c53fdb23"
Sep 10 20:57:47.567869 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:47.567829    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:57:47.568374 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:47.567964    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpwwj" podUID="97ae132c-459a-4cba-bb9f-822279e84225"
Sep 10 20:57:47.568477 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:47.568436    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:47.568617 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:47.568597    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r2gd" podUID="278ac5cb-23c6-4748-b751-d5f3c53fdb23"
Sep 10 20:57:47.568743 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:47.568717    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:47.568866 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:47.568842    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29b5p" podUID="8cefd1c8-a812-4cde-8fda-cc35af6e9833"
Sep 10 20:57:49.127484 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:49.127444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs\") pod \"network-metrics-daemon-4r2gd\" (UID: \"278ac5cb-23c6-4748-b751-d5f3c53fdb23\") " pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:49.127950 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:49.127614    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:49.127950 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:49.127686    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs podName:278ac5cb-23c6-4748-b751-d5f3c53fdb23 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:57.127665027 +0000 UTC m=+18.103325071 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs") pod "network-metrics-daemon-4r2gd" (UID: "278ac5cb-23c6-4748-b751-d5f3c53fdb23") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:49.228733 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:49.228699    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret\") pod \"global-pull-secret-syncer-29b5p\" (UID: \"8cefd1c8-a812-4cde-8fda-cc35af6e9833\") " pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:49.228928 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:49.228783    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc55n\" (UniqueName: \"kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n\") pod \"network-check-target-lpwwj\" (UID: \"97ae132c-459a-4cba-bb9f-822279e84225\") " pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:57:49.228928 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:49.228886    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:49.228928 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:49.228919    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:57:49.229113 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:49.228939    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:57:49.229113 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:49.228953    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qc55n for pod openshift-network-diagnostics/network-check-target-lpwwj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:49.229113 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:49.228970    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret podName:8cefd1c8-a812-4cde-8fda-cc35af6e9833 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:57.228947537 +0000 UTC m=+18.204607564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret") pod "global-pull-secret-syncer-29b5p" (UID: "8cefd1c8-a812-4cde-8fda-cc35af6e9833") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:49.229113 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:49.229005    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n podName:97ae132c-459a-4cba-bb9f-822279e84225 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:57.228987153 +0000 UTC m=+18.204647196 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qc55n" (UniqueName: "kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n") pod "network-check-target-lpwwj" (UID: "97ae132c-459a-4cba-bb9f-822279e84225") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:49.569460 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:49.569297    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:49.569460 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:49.569342    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:49.569696 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:49.569297    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:57:49.569696 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:49.569508    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29b5p" podUID="8cefd1c8-a812-4cde-8fda-cc35af6e9833"
Sep 10 20:57:49.569696 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:49.569603    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpwwj" podUID="97ae132c-459a-4cba-bb9f-822279e84225"
Sep 10 20:57:49.569696 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:49.569433    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r2gd" podUID="278ac5cb-23c6-4748-b751-d5f3c53fdb23"
Sep 10 20:57:51.568239 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:51.568194    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:57:51.568668 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:51.568335    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpwwj" podUID="97ae132c-459a-4cba-bb9f-822279e84225"
Sep 10 20:57:51.568733 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:51.568710    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:51.568894 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:51.568817    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r2gd" podUID="278ac5cb-23c6-4748-b751-d5f3c53fdb23"
Sep 10 20:57:51.568987 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:51.568959    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:51.569090 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:51.569055    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29b5p" podUID="8cefd1c8-a812-4cde-8fda-cc35af6e9833"
Sep 10 20:57:53.567740 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:53.567493    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:57:53.568169 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:53.567502    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:53.568169 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:53.567835    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpwwj" podUID="97ae132c-459a-4cba-bb9f-822279e84225"
Sep 10 20:57:53.568169 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:53.567557    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:53.568169 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:53.567941    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r2gd" podUID="278ac5cb-23c6-4748-b751-d5f3c53fdb23"
Sep 10 20:57:53.568169 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:53.568038    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29b5p" podUID="8cefd1c8-a812-4cde-8fda-cc35af6e9833"
Sep 10 20:57:55.568211 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:55.568155    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:55.568211 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:55.568155    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:57:55.568646 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:55.568169    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:55.568646 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:55.568307    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29b5p" podUID="8cefd1c8-a812-4cde-8fda-cc35af6e9833"
Sep 10 20:57:55.568646 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:55.568386    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpwwj" podUID="97ae132c-459a-4cba-bb9f-822279e84225"
Sep 10 20:57:55.568646 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:55.568491    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r2gd" podUID="278ac5cb-23c6-4748-b751-d5f3c53fdb23"
Sep 10 20:57:57.189473 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:57.189439    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs\") pod \"network-metrics-daemon-4r2gd\" (UID: \"278ac5cb-23c6-4748-b751-d5f3c53fdb23\") " pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:57.189932 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:57.189601    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:57.189932 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:57.189667    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs podName:278ac5cb-23c6-4748-b751-d5f3c53fdb23 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:13.189648865 +0000 UTC m=+34.165308902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs") pod "network-metrics-daemon-4r2gd" (UID: "278ac5cb-23c6-4748-b751-d5f3c53fdb23") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:57.290548 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:57.290515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc55n\" (UniqueName: \"kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n\") pod \"network-check-target-lpwwj\" (UID: \"97ae132c-459a-4cba-bb9f-822279e84225\") " pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:57:57.290696 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:57.290568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret\") pod \"global-pull-secret-syncer-29b5p\" (UID: \"8cefd1c8-a812-4cde-8fda-cc35af6e9833\") " pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:57.290696 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:57.290658    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:57:57.290696 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:57.290684    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:57:57.290696 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:57.290692    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:57.290696 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:57.290698    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qc55n for pod openshift-network-diagnostics/network-check-target-lpwwj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:57.290952 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:57.290748    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret podName:8cefd1c8-a812-4cde-8fda-cc35af6e9833 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:13.290732063 +0000 UTC m=+34.266392107 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret") pod "global-pull-secret-syncer-29b5p" (UID: "8cefd1c8-a812-4cde-8fda-cc35af6e9833") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:57.290952 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:57.290761    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n podName:97ae132c-459a-4cba-bb9f-822279e84225 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:13.290755048 +0000 UTC m=+34.266415073 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qc55n" (UniqueName: "kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n") pod "network-check-target-lpwwj" (UID: "97ae132c-459a-4cba-bb9f-822279e84225") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:57.571244 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:57.571151    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:57.571406 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:57.571151    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:57.571406 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:57.571151    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:57:57.571406 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:57.571379    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29b5p" podUID="8cefd1c8-a812-4cde-8fda-cc35af6e9833"
Sep 10 20:57:57.571538 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:57.571436    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpwwj" podUID="97ae132c-459a-4cba-bb9f-822279e84225"
Sep 10 20:57:57.571538 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:57.571284    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r2gd" podUID="278ac5cb-23c6-4748-b751-d5f3c53fdb23"
Sep 10 20:57:59.569035 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:59.569007    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:57:59.569517 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:59.569101    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpwwj" podUID="97ae132c-459a-4cba-bb9f-822279e84225"
Sep 10 20:57:59.569517 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:59.569189    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:57:59.569517 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:57:59.569232    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:57:59.569517 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:59.569317    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r2gd" podUID="278ac5cb-23c6-4748-b751-d5f3c53fdb23"
Sep 10 20:57:59.569517 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:57:59.569380    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29b5p" podUID="8cefd1c8-a812-4cde-8fda-cc35af6e9833"
Sep 10 20:58:01.570800 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.570642    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:58:01.571379 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.570655    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:58:01.571379 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:01.570871    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29b5p" podUID="8cefd1c8-a812-4cde-8fda-cc35af6e9833"
Sep 10 20:58:01.571379 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.570655    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:58:01.571379 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:01.570947    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpwwj" podUID="97ae132c-459a-4cba-bb9f-822279e84225"
Sep 10 20:58:01.571379 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:01.570986    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r2gd" podUID="278ac5cb-23c6-4748-b751-d5f3c53fdb23"
Sep 10 20:58:01.764728 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.764698    2581 generic.go:356] "Generic (PLEG): container finished" podID="1ee23d1e-61de-48f1-85c6-6d1ef3437ba2" containerID="5d0b5c65e5727c1ce0008fd696599a69aba53bd89d19ea5142f3df7b6cc6469d" exitCode=0
Sep 10 20:58:01.764881 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.764785    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qg95l" event={"ID":"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2","Type":"ContainerDied","Data":"5d0b5c65e5727c1ce0008fd696599a69aba53bd89d19ea5142f3df7b6cc6469d"}
Sep 10 20:58:01.766511 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.766483    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j5bg8" event={"ID":"59e8332d-e77e-4229-82a5-e3cf8ac84d81","Type":"ContainerStarted","Data":"106199be0da6b95c3baf4132499d91c8cf76b8b69d3c0420fd1954065a07f4fd"}
Sep 10 20:58:01.767912 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.767889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v6rlb" event={"ID":"a82c7020-9c1d-41a8-97a5-a752e87fba5a","Type":"ContainerStarted","Data":"2e94c69b292922e7acd01f06b8cc48ff9dec628c52b454a6b5ce56db57173b36"}
Sep 10 20:58:01.769346 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.769326    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ps8fr" event={"ID":"ddb8643c-235c-4694-b9dd-cd72d8986e42","Type":"ContainerStarted","Data":"bdf6d647e17583eaf3cb088d15c5700f40975e23d2312c8e639f327782feca8a"}
Sep 10 20:58:01.771025 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.770995    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ccgh9" event={"ID":"27105194-1643-4b6a-ae7b-2e0d76967bb6","Type":"ContainerStarted","Data":"b957a1d6e7eb5f6592810ac899b6ee06052843b6367b175b479a4ea7dfd57b03"}
Sep 10 20:58:01.772494 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.772461    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qp845" event={"ID":"cf7c2283-19e2-435f-81bd-2645fd914c77","Type":"ContainerStarted","Data":"c2a8274643dc53af901d54aca5caf8e54707920d2f838cbfad00aac66c913234"}
Sep 10 20:58:01.775593 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.775542    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m777h" event={"ID":"ab02629e-1b6c-4937-bf8e-26f31f42ed5b","Type":"ContainerStarted","Data":"802659c6d0d591e9cb18d3774d3641d8bbae925b0b9363077da536270f501503"}
Sep 10 20:58:01.775680 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.775616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m777h" event={"ID":"ab02629e-1b6c-4937-bf8e-26f31f42ed5b","Type":"ContainerStarted","Data":"e9f787ad72aad375a33ab9354bd48cfaad5673016cb9b572a93677a94123a156"}
Sep 10 20:58:01.775680 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.775669    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m777h" event={"ID":"ab02629e-1b6c-4937-bf8e-26f31f42ed5b","Type":"ContainerStarted","Data":"99a1d26c45f8e0db30255b8c38ac04fbc0f191bd5df95bad186d7c21889b44c9"}
Sep 10 20:58:01.775784 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.775716    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m777h" event={"ID":"ab02629e-1b6c-4937-bf8e-26f31f42ed5b","Type":"ContainerStarted","Data":"5c89902039f33840edd9468e1d60fdf5bfb85e3b420d74c90de2e9a22e8439d2"}
Sep 10 20:58:01.775784 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.775763    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m777h" event={"ID":"ab02629e-1b6c-4937-bf8e-26f31f42ed5b","Type":"ContainerStarted","Data":"1e4639c866836ab4591aff6fd807d81479d5b712d8714c65b10eb2e9ac654f2b"}
Sep 10 20:58:01.775875 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.775810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m777h" event={"ID":"ab02629e-1b6c-4937-bf8e-26f31f42ed5b","Type":"ContainerStarted","Data":"ba75db69301188f644a0a1cf8a7fc573d48a74878e8d9c38bf9e73cf3f2cf9a8"}
Sep 10 20:58:01.776916 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.776898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87" event={"ID":"f8007613-a73a-44e4-8507-e989280f250a","Type":"ContainerStarted","Data":"0f1fc447a403de1a402ee7608487e896941c1b00b533157875e3658563096000"}
Sep 10 20:58:01.792732 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.792694    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-6.ec2.internal" podStartSLOduration=20.792682655 podStartE2EDuration="20.792682655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:44.747122263 +0000 UTC m=+5.722782315" watchObservedRunningTime="2026-09-10 20:58:01.792682655 +0000 UTC m=+22.768342701"
Sep 10 20:58:01.859190 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.859152    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ccgh9" podStartSLOduration=2.514541958 podStartE2EDuration="20.859138694s" totalImagesPullingTime="18.344596736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:01.817164145 +0000 UTC m=+22.792824192" watchObservedRunningTime="2026-09-10 20:58:01.859138694 +0000 UTC m=+22.834798790"
Sep 10 20:58:01.884234 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.884175    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-j5bg8" podStartSLOduration=2.4966692569999998 podStartE2EDuration="20.884161735s" totalImagesPullingTime="18.387492478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:01.859395977 +0000 UTC m=+22.835056023" watchObservedRunningTime="2026-09-10 20:58:01.884161735 +0000 UTC m=+22.859821781"
Sep 10 20:58:01.918501 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.918453    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v6rlb" podStartSLOduration=10.807159072 podStartE2EDuration="21.918437873s" totalImagesPullingTime="11.111278801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:01.88481428 +0000 UTC m=+22.860474326" watchObservedRunningTime="2026-09-10 20:58:01.918437873 +0000 UTC m=+22.894097918"
Sep 10 20:58:01.918653 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.918589    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qp845" podStartSLOduration=3.571502162 podStartE2EDuration="21.918583037s" totalImagesPullingTime="18.347080875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:01.918164941 +0000 UTC m=+22.893824989" watchObservedRunningTime="2026-09-10 20:58:01.918583037 +0000 UTC m=+22.894243083"
Sep 10 20:58:01.948656 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:01.948615    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ps8fr" podStartSLOduration=3.607174674 podStartE2EDuration="21.948598943s" totalImagesPullingTime="18.341424269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:01.948041172 +0000 UTC m=+22.923701219" watchObservedRunningTime="2026-09-10 20:58:01.948598943 +0000 UTC m=+22.924258990"
Sep 10 20:58:02.432701 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:02.432669    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v6rlb"
Sep 10 20:58:02.433290 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:02.433261    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v6rlb"
Sep 10 20:58:02.780931 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:02.780852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dnjhl" event={"ID":"38d7a53b-34ee-421c-9e3e-3fafd56dcc09","Type":"ContainerStarted","Data":"5217273acbce621c37317b864ac06da457982292eb34f42537bf3d974b0f7c8a"}
Sep 10 20:58:02.781428 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:02.781081    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v6rlb"
Sep 10 20:58:02.781661 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:02.781645    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v6rlb"
Sep 10 20:58:02.804105 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:02.804064    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dnjhl" podStartSLOduration=3.471395081 podStartE2EDuration="21.80404887s" totalImagesPullingTime="18.332653789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:02.80376051 +0000 UTC m=+23.779420559" watchObservedRunningTime="2026-09-10 20:58:02.80404887 +0000 UTC m=+23.779708918"
Sep 10 20:58:02.888232 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:02.888194    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 20:58:03.526585 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:03.526418    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T20:58:02.888228212Z","UUID":"8a316924-f9ed-4d5c-873e-daa39862406d","Handler":null,"Name":"","Endpoint":""}
Sep 10 20:58:03.528246 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:03.528226    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 20:58:03.528382 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:03.528255    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 20:58:03.568162 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:03.568122    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:58:03.568162 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:03.568162    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:58:03.568358 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:03.568268    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r2gd" podUID="278ac5cb-23c6-4748-b751-d5f3c53fdb23"
Sep 10 20:58:03.568594 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:03.568538    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:58:03.568685 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:03.568651    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpwwj" podUID="97ae132c-459a-4cba-bb9f-822279e84225"
Sep 10 20:58:03.568732 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:03.568711    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29b5p" podUID="8cefd1c8-a812-4cde-8fda-cc35af6e9833"
Sep 10 20:58:03.784863 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:03.784839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87" event={"ID":"f8007613-a73a-44e4-8507-e989280f250a","Type":"ContainerStarted","Data":"860135accf5a34a6e7e22a43d29a33125a99bb41cab66f6927f197eeb4556d07"}
Sep 10 20:58:04.793187 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:04.793151    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m777h" event={"ID":"ab02629e-1b6c-4937-bf8e-26f31f42ed5b","Type":"ContainerStarted","Data":"018e9a919e4bbeff3a399c6b96c7ab8c928c5d77b91dad436db6741c4cda8393"}
Sep 10 20:58:04.795517 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:04.795491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87" event={"ID":"f8007613-a73a-44e4-8507-e989280f250a","Type":"ContainerStarted","Data":"3f4a5b6448c44cdc3b024e874410ed2e99efcd3485e32fd5b8b840f37e71691c"}
Sep 10 20:58:04.828716 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:04.828676    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhq87" podStartSLOduration=3.997439219 podStartE2EDuration="24.828657405s" totalImagesPullingTime="20.831218186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:04.828041997 +0000 UTC m=+25.803702046" watchObservedRunningTime="2026-09-10 20:58:04.828657405 +0000 UTC m=+25.804317452"
Sep 10 20:58:05.571236 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:05.571193    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:58:05.571443 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:05.571307    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:58:05.571443 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:05.571319    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29b5p" podUID="8cefd1c8-a812-4cde-8fda-cc35af6e9833"
Sep 10 20:58:05.571443 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:05.571332    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:58:05.571443 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:05.571404    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r2gd" podUID="278ac5cb-23c6-4748-b751-d5f3c53fdb23"
Sep 10 20:58:05.571645 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:05.571468    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpwwj" podUID="97ae132c-459a-4cba-bb9f-822279e84225"
Sep 10 20:58:07.570760 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:07.570581    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:58:07.571290 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:07.570588    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:58:07.571290 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:07.570833    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29b5p" podUID="8cefd1c8-a812-4cde-8fda-cc35af6e9833"
Sep 10 20:58:07.571290 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:07.570588    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:58:07.571290 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:07.570914    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r2gd" podUID="278ac5cb-23c6-4748-b751-d5f3c53fdb23"
Sep 10 20:58:07.571290 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:07.570968    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpwwj" podUID="97ae132c-459a-4cba-bb9f-822279e84225"
Sep 10 20:58:07.803906 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:07.803870    2581 generic.go:356] "Generic (PLEG): container finished" podID="1ee23d1e-61de-48f1-85c6-6d1ef3437ba2" containerID="78d661811049a60a6affb2c6c7b444d1ca9fa6e3d9b4d542f0a99f4bae5b63d5" exitCode=0
Sep 10 20:58:07.804105 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:07.803955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qg95l" event={"ID":"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2","Type":"ContainerDied","Data":"78d661811049a60a6affb2c6c7b444d1ca9fa6e3d9b4d542f0a99f4bae5b63d5"}
Sep 10 20:58:07.808011 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:07.807991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m777h" event={"ID":"ab02629e-1b6c-4937-bf8e-26f31f42ed5b","Type":"ContainerStarted","Data":"b332b8bbb444a764a944d044c4d3666f20944eef1f4e71330e043f5b16043d1b"}
Sep 10 20:58:07.808251 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:07.808234    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:58:07.808320 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:07.808306    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:58:07.808380 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:07.808356    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:58:07.822669 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:07.822611    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:58:07.822793 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:07.822781    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:58:09.006051 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:09.005943    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m777h" podStartSLOduration=10.611814285 podStartE2EDuration="29.005925879s" totalImagesPullingTime="18.394111594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:07.885084724 +0000 UTC m=+28.860744771" watchObservedRunningTime="2026-09-10 20:58:09.005925879 +0000 UTC m=+29.981585926"
Sep 10 20:58:09.007015 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:09.006970    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-29b5p"]
Sep 10 20:58:09.007165 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:09.007150    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:58:09.007627 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:09.007318    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29b5p" podUID="8cefd1c8-a812-4cde-8fda-cc35af6e9833"
Sep 10 20:58:09.010005 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:09.009985    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lpwwj"]
Sep 10 20:58:09.010091 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:09.010061    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:58:09.010140 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:09.010123    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpwwj" podUID="97ae132c-459a-4cba-bb9f-822279e84225"
Sep 10 20:58:09.012430 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:09.012409    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4r2gd"]
Sep 10 20:58:09.012528 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:09.012516    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:58:09.012628 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:09.012605    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r2gd" podUID="278ac5cb-23c6-4748-b751-d5f3c53fdb23"
Sep 10 20:58:09.816040 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:09.814535    2581 generic.go:356] "Generic (PLEG): container finished" podID="1ee23d1e-61de-48f1-85c6-6d1ef3437ba2" containerID="35b8ec1cd29d43e154637cdf9d03ae88c78d67269909f828389d077feb34ac78" exitCode=0
Sep 10 20:58:09.816040 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:09.816034    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qg95l" event={"ID":"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2","Type":"ContainerDied","Data":"35b8ec1cd29d43e154637cdf9d03ae88c78d67269909f828389d077feb34ac78"}
Sep 10 20:58:10.567718 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:10.567554    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:58:10.568116 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:10.567603    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:58:10.568116 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:10.567821    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r2gd" podUID="278ac5cb-23c6-4748-b751-d5f3c53fdb23"
Sep 10 20:58:10.568116 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:10.567908    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29b5p" podUID="8cefd1c8-a812-4cde-8fda-cc35af6e9833"
Sep 10 20:58:10.568116 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:10.567621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:58:10.568116 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:10.568062    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpwwj" podUID="97ae132c-459a-4cba-bb9f-822279e84225"
Sep 10 20:58:11.820725 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:11.820694    2581 generic.go:356] "Generic (PLEG): container finished" podID="1ee23d1e-61de-48f1-85c6-6d1ef3437ba2" containerID="2d3f2728099ed11af2228c04382221a7cc8f31cf0e56b09eb2f301aec6495685" exitCode=0
Sep 10 20:58:11.820725 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:11.820729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qg95l" event={"ID":"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2","Type":"ContainerDied","Data":"2d3f2728099ed11af2228c04382221a7cc8f31cf0e56b09eb2f301aec6495685"}
Sep 10 20:58:12.567736 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:12.567705    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:58:12.567911 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:12.567745    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:58:12.567911 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:12.567748    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:58:12.567911 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:12.567837    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29b5p" podUID="8cefd1c8-a812-4cde-8fda-cc35af6e9833"
Sep 10 20:58:12.568069 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:12.567923    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r2gd" podUID="278ac5cb-23c6-4748-b751-d5f3c53fdb23"
Sep 10 20:58:12.568069 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:12.567989    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpwwj" podUID="97ae132c-459a-4cba-bb9f-822279e84225"
Sep 10 20:58:13.209974 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.209941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs\") pod \"network-metrics-daemon-4r2gd\" (UID: \"278ac5cb-23c6-4748-b751-d5f3c53fdb23\") " pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:58:13.210436 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:13.210074    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:13.210436 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:13.210127    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs podName:278ac5cb-23c6-4748-b751-d5f3c53fdb23 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:45.210110973 +0000 UTC m=+66.185770998 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs") pod "network-metrics-daemon-4r2gd" (UID: "278ac5cb-23c6-4748-b751-d5f3c53fdb23") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:13.310946 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.310913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc55n\" (UniqueName: \"kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n\") pod \"network-check-target-lpwwj\" (UID: \"97ae132c-459a-4cba-bb9f-822279e84225\") " pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:58:13.311111 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.310961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret\") pod \"global-pull-secret-syncer-29b5p\" (UID: \"8cefd1c8-a812-4cde-8fda-cc35af6e9833\") " pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:58:13.311111 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:13.311100    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:13.311257 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:13.311170    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret podName:8cefd1c8-a812-4cde-8fda-cc35af6e9833 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:45.311151098 +0000 UTC m=+66.286811133 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret") pod "global-pull-secret-syncer-29b5p" (UID: "8cefd1c8-a812-4cde-8fda-cc35af6e9833") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:13.311257 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:13.311103    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:58:13.311257 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:13.311245    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:58:13.311257 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:13.311259    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qc55n for pod openshift-network-diagnostics/network-check-target-lpwwj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:13.311450 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:13.311317    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n podName:97ae132c-459a-4cba-bb9f-822279e84225 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:45.311300618 +0000 UTC m=+66.286960647 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qc55n" (UniqueName: "kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n") pod "network-check-target-lpwwj" (UID: "97ae132c-459a-4cba-bb9f-822279e84225") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:13.796575 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.796549    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-6.ec2.internal" event="NodeReady"
Sep 10 20:58:13.796727 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.796657    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 20:58:13.854342 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.854314    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f469fbc67-26fpg"]
Sep 10 20:58:13.868304 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.868275    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:13.871228 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.871182    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-c5cks"]
Sep 10 20:58:13.871677 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.871656    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 20:58:13.871797 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.871656    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 20:58:13.871965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.871948    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8bbrk\""
Sep 10 20:58:13.872128 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.872102    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 20:58:13.877226 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.877195    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 20:58:13.882386 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.882369    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c5cks"
Sep 10 20:58:13.886421 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.886404    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f469fbc67-26fpg"]
Sep 10 20:58:13.887212 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.887182    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7mplf"]
Sep 10 20:58:13.893232 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.893135    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 20:58:13.893355 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.893244    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-278wb\""
Sep 10 20:58:13.893438 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.893424    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 20:58:13.902402 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.902384    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 20:58:13.902813 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.902796    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c5cks"]
Sep 10 20:58:13.902888 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.902878    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:13.905427 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.905407    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 20:58:13.905657 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.905641    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9hkk2\""
Sep 10 20:58:13.909742 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.909723    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 20:58:13.915900 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:13.915880    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7mplf"]
Sep 10 20:58:14.014797 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.014768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgkgw\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-kube-api-access-lgkgw\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.014940 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.014799    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c25e03f-53c0-43c0-ad21-7301f930243f-config-volume\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:14.014940 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.014838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd68b468-3b12-4738-b092-1f3e6e3c3356-ca-trust-extracted\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.014940 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.014857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-certificates\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.014940 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.014872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd68b468-3b12-4738-b092-1f3e6e3c3356-installation-pull-secrets\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.014940 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.014889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:14.014940 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.014917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n77q\" (UniqueName: \"kubernetes.io/projected/6d565bb5-5969-45f9-b862-58ccf6248c9a-kube-api-access-7n77q\") pod \"ingress-canary-c5cks\" (UID: \"6d565bb5-5969-45f9-b862-58ccf6248c9a\") " pod="openshift-ingress-canary/ingress-canary-c5cks"
Sep 10 20:58:14.014940 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.014939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd68b468-3b12-4738-b092-1f3e6e3c3356-trusted-ca\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.015148 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.014960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlzj4\" (UniqueName: \"kubernetes.io/projected/9c25e03f-53c0-43c0-ad21-7301f930243f-kube-api-access-vlzj4\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:14.015148 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.014981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c25e03f-53c0-43c0-ad21-7301f930243f-tmp-dir\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:14.015148 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.015006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-bound-sa-token\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.015148 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.015069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.015148 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.015107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd68b468-3b12-4738-b092-1f3e6e3c3356-image-registry-private-configuration\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.015148 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.015123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d565bb5-5969-45f9-b862-58ccf6248c9a-cert\") pod \"ingress-canary-c5cks\" (UID: \"6d565bb5-5969-45f9-b862-58ccf6248c9a\") " pod="openshift-ingress-canary/ingress-canary-c5cks"
Sep 10 20:58:14.115637 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.115597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.115826 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.115673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd68b468-3b12-4738-b092-1f3e6e3c3356-image-registry-private-configuration\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.115826 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.115697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d565bb5-5969-45f9-b862-58ccf6248c9a-cert\") pod \"ingress-canary-c5cks\" (UID: \"6d565bb5-5969-45f9-b862-58ccf6248c9a\") " pod="openshift-ingress-canary/ingress-canary-c5cks"
Sep 10 20:58:14.115826 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.115725    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgkgw\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-kube-api-access-lgkgw\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.115826 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:14.115747    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:58:14.115826 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:14.115769    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f469fbc67-26fpg: secret "image-registry-tls" not found
Sep 10 20:58:14.115826 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:14.115823    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls podName:dd68b468-3b12-4738-b092-1f3e6e3c3356 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:14.615808905 +0000 UTC m=+35.591469000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls") pod "image-registry-f469fbc67-26fpg" (UID: "dd68b468-3b12-4738-b092-1f3e6e3c3356") : secret "image-registry-tls" not found
Sep 10 20:58:14.116106 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.115750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c25e03f-53c0-43c0-ad21-7301f930243f-config-volume\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:14.116106 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.116062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd68b468-3b12-4738-b092-1f3e6e3c3356-ca-trust-extracted\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.116106 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.116084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-certificates\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.116106 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.116103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd68b468-3b12-4738-b092-1f3e6e3c3356-installation-pull-secrets\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.116298 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.116119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:14.116298 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.116140    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n77q\" (UniqueName: \"kubernetes.io/projected/6d565bb5-5969-45f9-b862-58ccf6248c9a-kube-api-access-7n77q\") pod \"ingress-canary-c5cks\" (UID: \"6d565bb5-5969-45f9-b862-58ccf6248c9a\") " pod="openshift-ingress-canary/ingress-canary-c5cks"
Sep 10 20:58:14.116298 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.116154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd68b468-3b12-4738-b092-1f3e6e3c3356-trusted-ca\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.116298 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.116180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlzj4\" (UniqueName: \"kubernetes.io/projected/9c25e03f-53c0-43c0-ad21-7301f930243f-kube-api-access-vlzj4\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:14.116298 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.116219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c25e03f-53c0-43c0-ad21-7301f930243f-tmp-dir\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:14.116298 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.116237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-bound-sa-token\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.116573 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.116322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c25e03f-53c0-43c0-ad21-7301f930243f-config-volume\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:14.116573 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:14.116507    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:14.116654 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:14.116585    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls podName:9c25e03f-53c0-43c0-ad21-7301f930243f nodeName:}" failed. No retries permitted until 2026-09-10 20:58:14.616568849 +0000 UTC m=+35.592228882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls") pod "dns-default-7mplf" (UID: "9c25e03f-53c0-43c0-ad21-7301f930243f") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:14.116834 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.116812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c25e03f-53c0-43c0-ad21-7301f930243f-tmp-dir\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:14.117144 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.117095    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-certificates\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.117275 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.117222    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd68b468-3b12-4738-b092-1f3e6e3c3356-ca-trust-extracted\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.117458 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.117425    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd68b468-3b12-4738-b092-1f3e6e3c3356-trusted-ca\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.120487 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.120433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd68b468-3b12-4738-b092-1f3e6e3c3356-image-registry-private-configuration\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.120487 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.120451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d565bb5-5969-45f9-b862-58ccf6248c9a-cert\") pod \"ingress-canary-c5cks\" (UID: \"6d565bb5-5969-45f9-b862-58ccf6248c9a\") " pod="openshift-ingress-canary/ingress-canary-c5cks"
Sep 10 20:58:14.120487 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.120440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd68b468-3b12-4738-b092-1f3e6e3c3356-installation-pull-secrets\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.131243 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.131221    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n77q\" (UniqueName: \"kubernetes.io/projected/6d565bb5-5969-45f9-b862-58ccf6248c9a-kube-api-access-7n77q\") pod \"ingress-canary-c5cks\" (UID: \"6d565bb5-5969-45f9-b862-58ccf6248c9a\") " pod="openshift-ingress-canary/ingress-canary-c5cks"
Sep 10 20:58:14.132421 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.132399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-bound-sa-token\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.133944 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.133914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlzj4\" (UniqueName: \"kubernetes.io/projected/9c25e03f-53c0-43c0-ad21-7301f930243f-kube-api-access-vlzj4\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:14.134903 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.134886    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgkgw\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-kube-api-access-lgkgw\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.194027 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.193955    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c5cks"
Sep 10 20:58:14.523291 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.523066    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c5cks"]
Sep 10 20:58:14.526801 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:58:14.526778    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d565bb5_5969_45f9_b862_58ccf6248c9a.slice/crio-bba4f044cdec1b228ca8b4903ea76295602e092f3451bb51c68f5258bfd3e9c9 WatchSource:0}: Error finding container bba4f044cdec1b228ca8b4903ea76295602e092f3451bb51c68f5258bfd3e9c9: Status 404 returned error can't find the container with id bba4f044cdec1b228ca8b4903ea76295602e092f3451bb51c68f5258bfd3e9c9
Sep 10 20:58:14.567604 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.567581    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:58:14.567730 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.567668    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:58:14.567730 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.567697    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:58:14.570934 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.570915    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ll8tt\""
Sep 10 20:58:14.571041 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.570918    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 20:58:14.571625 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.571607    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 20:58:14.571693 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.571638    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 20:58:14.571693 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.571655    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 20:58:14.571693 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.571661    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tjcr5\""
Sep 10 20:58:14.621762 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.621730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:14.621890 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:14.621856    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:58:14.621890 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:14.621870    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f469fbc67-26fpg: secret "image-registry-tls" not found
Sep 10 20:58:14.622033 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.621909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:14.622033 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:14.621921    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls podName:dd68b468-3b12-4738-b092-1f3e6e3c3356 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:15.621903309 +0000 UTC m=+36.597563351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls") pod "image-registry-f469fbc67-26fpg" (UID: "dd68b468-3b12-4738-b092-1f3e6e3c3356") : secret "image-registry-tls" not found
Sep 10 20:58:14.622033 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:14.622025    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:14.622226 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:14.622077    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls podName:9c25e03f-53c0-43c0-ad21-7301f930243f nodeName:}" failed. No retries permitted until 2026-09-10 20:58:15.622060143 +0000 UTC m=+36.597720184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls") pod "dns-default-7mplf" (UID: "9c25e03f-53c0-43c0-ad21-7301f930243f") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:14.828374 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:14.828344    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c5cks" event={"ID":"6d565bb5-5969-45f9-b862-58ccf6248c9a","Type":"ContainerStarted","Data":"bba4f044cdec1b228ca8b4903ea76295602e092f3451bb51c68f5258bfd3e9c9"}
Sep 10 20:58:15.629148 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:15.629112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:15.629730 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:15.629194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:15.629730 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:15.629283    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:58:15.629730 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:15.629300    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f469fbc67-26fpg: secret "image-registry-tls" not found
Sep 10 20:58:15.629730 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:15.629302    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:15.629730 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:15.629361    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls podName:dd68b468-3b12-4738-b092-1f3e6e3c3356 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:17.629342957 +0000 UTC m=+38.605002992 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls") pod "image-registry-f469fbc67-26fpg" (UID: "dd68b468-3b12-4738-b092-1f3e6e3c3356") : secret "image-registry-tls" not found
Sep 10 20:58:15.629730 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:15.629379    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls podName:9c25e03f-53c0-43c0-ad21-7301f930243f nodeName:}" failed. No retries permitted until 2026-09-10 20:58:17.629370423 +0000 UTC m=+38.605030453 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls") pod "dns-default-7mplf" (UID: "9c25e03f-53c0-43c0-ad21-7301f930243f") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:17.645449 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:17.645423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:17.645854 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:17.645480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:17.645854 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:17.645568    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:17.645854 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:17.645569    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:58:17.645854 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:17.645587    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f469fbc67-26fpg: secret "image-registry-tls" not found
Sep 10 20:58:17.645854 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:17.645617    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls podName:9c25e03f-53c0-43c0-ad21-7301f930243f nodeName:}" failed. No retries permitted until 2026-09-10 20:58:21.645603748 +0000 UTC m=+42.621263776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls") pod "dns-default-7mplf" (UID: "9c25e03f-53c0-43c0-ad21-7301f930243f") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:17.645854 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:17.645630    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls podName:dd68b468-3b12-4738-b092-1f3e6e3c3356 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:21.645624123 +0000 UTC m=+42.621284148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls") pod "image-registry-f469fbc67-26fpg" (UID: "dd68b468-3b12-4738-b092-1f3e6e3c3356") : secret "image-registry-tls" not found
Sep 10 20:58:17.838836 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:17.838810    2581 generic.go:356] "Generic (PLEG): container finished" podID="1ee23d1e-61de-48f1-85c6-6d1ef3437ba2" containerID="4d6d5aff7f3ffa2adaebb45dc508e6ac428e92b09d78237fa3a431df338efc80" exitCode=0
Sep 10 20:58:17.838965 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:17.838869    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qg95l" event={"ID":"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2","Type":"ContainerDied","Data":"4d6d5aff7f3ffa2adaebb45dc508e6ac428e92b09d78237fa3a431df338efc80"}
Sep 10 20:58:18.844113 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:18.844083    2581 generic.go:356] "Generic (PLEG): container finished" podID="1ee23d1e-61de-48f1-85c6-6d1ef3437ba2" containerID="239e22bf2cefd408f3156ab2095fcc701a4d4021634dfee6165f391b6e28c59b" exitCode=0
Sep 10 20:58:18.844599 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:18.844159    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qg95l" event={"ID":"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2","Type":"ContainerDied","Data":"239e22bf2cefd408f3156ab2095fcc701a4d4021634dfee6165f391b6e28c59b"}
Sep 10 20:58:18.845651 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:18.845630    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c5cks" event={"ID":"6d565bb5-5969-45f9-b862-58ccf6248c9a","Type":"ContainerStarted","Data":"2ad0a469bfb0b758e46dc90dd19517e6b1be9aeda67cc3a5e363c251c6984238"}
Sep 10 20:58:18.911877 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:18.911829    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-c5cks" podStartSLOduration=2.704252779 podStartE2EDuration="5.911812035s" totalImagesPullingTime="3.207559256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:18.9109519 +0000 UTC m=+39.886611950" watchObservedRunningTime="2026-09-10 20:58:18.911812035 +0000 UTC m=+39.887472079"
Sep 10 20:58:19.851416 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:19.851375    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qg95l" event={"ID":"1ee23d1e-61de-48f1-85c6-6d1ef3437ba2","Type":"ContainerStarted","Data":"31fc97bfbba05277d7cf1081c60f79932b7e2ca23166f90b7dda4dea590d8ed0"}
Sep 10 20:58:21.675777 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:21.675743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:21.676182 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:21.675798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:21.676182 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:21.675891    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:21.676182 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:21.675897    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:58:21.676182 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:21.675917    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f469fbc67-26fpg: secret "image-registry-tls" not found
Sep 10 20:58:21.676182 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:21.675941    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls podName:9c25e03f-53c0-43c0-ad21-7301f930243f nodeName:}" failed. No retries permitted until 2026-09-10 20:58:29.675927627 +0000 UTC m=+50.651587657 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls") pod "dns-default-7mplf" (UID: "9c25e03f-53c0-43c0-ad21-7301f930243f") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:21.676182 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:21.675962    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls podName:dd68b468-3b12-4738-b092-1f3e6e3c3356 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:29.675949617 +0000 UTC m=+50.651609647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls") pod "image-registry-f469fbc67-26fpg" (UID: "dd68b468-3b12-4738-b092-1f3e6e3c3356") : secret "image-registry-tls" not found
Sep 10 20:58:29.729631 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:29.729597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:29.730000 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:29.729654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:29.730000 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:29.729737    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:29.730000 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:29.729742    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:58:29.730000 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:29.729759    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f469fbc67-26fpg: secret "image-registry-tls" not found
Sep 10 20:58:29.730000 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:29.729794    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls podName:9c25e03f-53c0-43c0-ad21-7301f930243f nodeName:}" failed. No retries permitted until 2026-09-10 20:58:45.729782377 +0000 UTC m=+66.705442401 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls") pod "dns-default-7mplf" (UID: "9c25e03f-53c0-43c0-ad21-7301f930243f") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:29.730000 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:29.729812    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls podName:dd68b468-3b12-4738-b092-1f3e6e3c3356 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:45.729798828 +0000 UTC m=+66.705458853 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls") pod "image-registry-f469fbc67-26fpg" (UID: "dd68b468-3b12-4738-b092-1f3e6e3c3356") : secret "image-registry-tls" not found
Sep 10 20:58:39.833415 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:39.833377    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m777h"
Sep 10 20:58:39.868720 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:39.868671    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qg95l" podStartSLOduration=27.513373173 podStartE2EDuration="58.868654657s" totalImagesPullingTime="31.355281484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:19.921674523 +0000 UTC m=+40.897334569" watchObservedRunningTime="2026-09-10 20:58:39.868654657 +0000 UTC m=+60.844314703"
Sep 10 20:58:40.592886 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:40.592860    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk"]
Sep 10 20:58:40.627244 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:40.627216    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk"]
Sep 10 20:58:40.627390 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:40.627330    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk"
Sep 10 20:58:40.630011 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:40.629986    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 20:58:40.630231 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:40.630195    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:58:40.630231 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:40.630225    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 20:58:40.630766 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:40.630751    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-g4p8s\""
Sep 10 20:58:40.704273 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:40.704243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ctgwk\" (UID: \"91ba5603-16c5-4fa8-b573-b57774494893\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk"
Sep 10 20:58:40.704273 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:40.704274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8pkg\" (UniqueName: \"kubernetes.io/projected/91ba5603-16c5-4fa8-b573-b57774494893-kube-api-access-g8pkg\") pod \"cluster-samples-operator-86499c89c4-ctgwk\" (UID: \"91ba5603-16c5-4fa8-b573-b57774494893\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk"
Sep 10 20:58:40.805177 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:40.805148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ctgwk\" (UID: \"91ba5603-16c5-4fa8-b573-b57774494893\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk"
Sep 10 20:58:40.805177 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:40.805181    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8pkg\" (UniqueName: \"kubernetes.io/projected/91ba5603-16c5-4fa8-b573-b57774494893-kube-api-access-g8pkg\") pod \"cluster-samples-operator-86499c89c4-ctgwk\" (UID: \"91ba5603-16c5-4fa8-b573-b57774494893\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk"
Sep 10 20:58:40.805482 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:40.805316    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 20:58:40.805482 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:40.805405    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls podName:91ba5603-16c5-4fa8-b573-b57774494893 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:41.305383577 +0000 UTC m=+62.281043624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-ctgwk" (UID: "91ba5603-16c5-4fa8-b573-b57774494893") : secret "samples-operator-tls" not found
Sep 10 20:58:40.816420 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:40.816393    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8pkg\" (UniqueName: \"kubernetes.io/projected/91ba5603-16c5-4fa8-b573-b57774494893-kube-api-access-g8pkg\") pod \"cluster-samples-operator-86499c89c4-ctgwk\" (UID: \"91ba5603-16c5-4fa8-b573-b57774494893\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk"
Sep 10 20:58:41.309149 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:41.309098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ctgwk\" (UID: \"91ba5603-16c5-4fa8-b573-b57774494893\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk"
Sep 10 20:58:41.309654 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:41.309290    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 20:58:41.309654 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:41.309374    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls podName:91ba5603-16c5-4fa8-b573-b57774494893 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:42.309352062 +0000 UTC m=+63.285012091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-ctgwk" (UID: "91ba5603-16c5-4fa8-b573-b57774494893") : secret "samples-operator-tls" not found
Sep 10 20:58:42.315599 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:42.315556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ctgwk\" (UID: \"91ba5603-16c5-4fa8-b573-b57774494893\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk"
Sep 10 20:58:42.316078 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:42.315691    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 20:58:42.316078 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:42.315748    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls podName:91ba5603-16c5-4fa8-b573-b57774494893 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:44.315731221 +0000 UTC m=+65.291391246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-ctgwk" (UID: "91ba5603-16c5-4fa8-b573-b57774494893") : secret "samples-operator-tls" not found
Sep 10 20:58:44.327125 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:44.327085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ctgwk\" (UID: \"91ba5603-16c5-4fa8-b573-b57774494893\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk"
Sep 10 20:58:44.327508 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:44.327226    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 20:58:44.327508 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:44.327300    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls podName:91ba5603-16c5-4fa8-b573-b57774494893 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:48.327285563 +0000 UTC m=+69.302945589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-ctgwk" (UID: "91ba5603-16c5-4fa8-b573-b57774494893") : secret "samples-operator-tls" not found
Sep 10 20:58:45.231337 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.231303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs\") pod \"network-metrics-daemon-4r2gd\" (UID: \"278ac5cb-23c6-4748-b751-d5f3c53fdb23\") " pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:58:45.233911 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.233893    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 20:58:45.241667 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:45.241649    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 20:58:45.241715 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:45.241703    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs podName:278ac5cb-23c6-4748-b751-d5f3c53fdb23 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:49.241686026 +0000 UTC m=+130.217346056 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs") pod "network-metrics-daemon-4r2gd" (UID: "278ac5cb-23c6-4748-b751-d5f3c53fdb23") : secret "metrics-daemon-secret" not found
Sep 10 20:58:45.331606 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.331578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc55n\" (UniqueName: \"kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n\") pod \"network-check-target-lpwwj\" (UID: \"97ae132c-459a-4cba-bb9f-822279e84225\") " pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:58:45.331881 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.331627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret\") pod \"global-pull-secret-syncer-29b5p\" (UID: \"8cefd1c8-a812-4cde-8fda-cc35af6e9833\") " pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:58:45.334633 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.334611    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 20:58:45.334633 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.334625    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 20:58:45.343942 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.343915    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cefd1c8-a812-4cde-8fda-cc35af6e9833-original-pull-secret\") pod \"global-pull-secret-syncer-29b5p\" (UID: \"8cefd1c8-a812-4cde-8fda-cc35af6e9833\") " pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:58:45.344428 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.344413    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 20:58:45.354474 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.354448    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc55n\" (UniqueName: \"kubernetes.io/projected/97ae132c-459a-4cba-bb9f-822279e84225-kube-api-access-qc55n\") pod \"network-check-target-lpwwj\" (UID: \"97ae132c-459a-4cba-bb9f-822279e84225\") " pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:58:45.478404 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.478378    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29b5p"
Sep 10 20:58:45.494503 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.494485    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ll8tt\""
Sep 10 20:58:45.499096 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.499072    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dph2p"]
Sep 10 20:58:45.502111 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.502092    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:58:45.502210 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.502118    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:45.507787 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.507752    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 20:58:45.507787 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.507766    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 20:58:45.507950 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.507753    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 20:58:45.507950 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.507752    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k7jqh\""
Sep 10 20:58:45.508742 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.508728    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 20:58:45.532558 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.532528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7fd3843-b239-4f87-ab91-808f4a31f064-data-volume\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:45.532642 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.532577    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7fd3843-b239-4f87-ab91-808f4a31f064-crio-socket\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:45.532642 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.532606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:45.532750 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.532635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvzfx\" (UniqueName: \"kubernetes.io/projected/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-api-access-vvzfx\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:45.532750 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.532734    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:45.540573 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.540551    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dph2p"]
Sep 10 20:58:45.633000 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.632966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:45.633140 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.633039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7fd3843-b239-4f87-ab91-808f4a31f064-data-volume\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:45.633140 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.633059    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7fd3843-b239-4f87-ab91-808f4a31f064-crio-socket\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:45.633140 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.633074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:45.633140 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:45.633090    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:45.633140 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.633091    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvzfx\" (UniqueName: \"kubernetes.io/projected/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-api-access-vvzfx\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:45.633378 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:45.633155    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls podName:e7fd3843-b239-4f87-ab91-808f4a31f064 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:46.133134152 +0000 UTC m=+67.108794194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dph2p" (UID: "e7fd3843-b239-4f87-ab91-808f4a31f064") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:45.633378 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.633156    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7fd3843-b239-4f87-ab91-808f4a31f064-crio-socket\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:45.633378 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:45.633179    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:58:45.633378 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:45.633271    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm podName:e7fd3843-b239-4f87-ab91-808f4a31f064 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:46.133252993 +0000 UTC m=+67.108913035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dph2p" (UID: "e7fd3843-b239-4f87-ab91-808f4a31f064") : configmap "kube-rbac-proxy" not found
Sep 10 20:58:45.633510 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.633432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7fd3843-b239-4f87-ab91-808f4a31f064-data-volume\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:45.641829 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.641813    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvzfx\" (UniqueName: \"kubernetes.io/projected/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-api-access-vvzfx\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:45.734092 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.734040    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:58:45.734173 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.734114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:58:45.734243 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:45.734189    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:58:45.734243 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:45.734221    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f469fbc67-26fpg: secret "image-registry-tls" not found
Sep 10 20:58:45.734314 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:45.734264    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:45.734314 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:45.734292    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls podName:dd68b468-3b12-4738-b092-1f3e6e3c3356 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:17.734277979 +0000 UTC m=+98.709938010 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls") pod "image-registry-f469fbc67-26fpg" (UID: "dd68b468-3b12-4738-b092-1f3e6e3c3356") : secret "image-registry-tls" not found
Sep 10 20:58:45.734314 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:45.734310    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls podName:9c25e03f-53c0-43c0-ad21-7301f930243f nodeName:}" failed. No retries permitted until 2026-09-10 20:59:17.734297882 +0000 UTC m=+98.709957915 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls") pod "dns-default-7mplf" (UID: "9c25e03f-53c0-43c0-ad21-7301f930243f") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:45.750610 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.750586    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-29b5p"]
Sep 10 20:58:45.755055 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:58:45.755017    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cefd1c8_a812_4cde_8fda_cc35af6e9833.slice/crio-17129edd758c88c79a4ebaa3f9a960f52ac5fe7a36c81b4219e045bffa0f6191 WatchSource:0}: Error finding container 17129edd758c88c79a4ebaa3f9a960f52ac5fe7a36c81b4219e045bffa0f6191: Status 404 returned error can't find the container with id 17129edd758c88c79a4ebaa3f9a960f52ac5fe7a36c81b4219e045bffa0f6191
Sep 10 20:58:45.792047 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.790699    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lpwwj"]
Sep 10 20:58:45.794566 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:58:45.794544    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ae132c_459a_4cba_bb9f_822279e84225.slice/crio-840f11bf93b6aeeaa50248563c1f22fab0976f6cb1f5eba9b03ee038665a7141 WatchSource:0}: Error finding container 840f11bf93b6aeeaa50248563c1f22fab0976f6cb1f5eba9b03ee038665a7141: Status 404 returned error can't find the container with id 840f11bf93b6aeeaa50248563c1f22fab0976f6cb1f5eba9b03ee038665a7141
Sep 10 20:58:45.912429 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.912400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lpwwj" event={"ID":"97ae132c-459a-4cba-bb9f-822279e84225","Type":"ContainerStarted","Data":"840f11bf93b6aeeaa50248563c1f22fab0976f6cb1f5eba9b03ee038665a7141"}
Sep 10 20:58:45.913434 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:45.913412    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-29b5p" event={"ID":"8cefd1c8-a812-4cde-8fda-cc35af6e9833","Type":"ContainerStarted","Data":"17129edd758c88c79a4ebaa3f9a960f52ac5fe7a36c81b4219e045bffa0f6191"}
Sep 10 20:58:46.136111 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:46.136075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:46.136287 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:46.136134    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:46.136287 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:46.136231    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:58:46.136287 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:46.136258    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:46.136287 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:46.136283    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm podName:e7fd3843-b239-4f87-ab91-808f4a31f064 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:47.136269871 +0000 UTC m=+68.111929897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dph2p" (UID: "e7fd3843-b239-4f87-ab91-808f4a31f064") : configmap "kube-rbac-proxy" not found
Sep 10 20:58:46.136423 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:46.136315    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls podName:e7fd3843-b239-4f87-ab91-808f4a31f064 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:47.136301568 +0000 UTC m=+68.111961593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dph2p" (UID: "e7fd3843-b239-4f87-ab91-808f4a31f064") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:47.145317 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:47.145247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:47.145786 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:47.145345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:47.145786 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:47.145494    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:47.145786 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:47.145558    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls podName:e7fd3843-b239-4f87-ab91-808f4a31f064 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:49.145538036 +0000 UTC m=+70.121198074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dph2p" (UID: "e7fd3843-b239-4f87-ab91-808f4a31f064") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:47.145786 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:47.145611    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:58:47.145786 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:47.145645    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm podName:e7fd3843-b239-4f87-ab91-808f4a31f064 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:49.14563526 +0000 UTC m=+70.121295288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dph2p" (UID: "e7fd3843-b239-4f87-ab91-808f4a31f064") : configmap "kube-rbac-proxy" not found
Sep 10 20:58:47.633483 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:47.633450    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ps8fr_ddb8643c-235c-4694-b9dd-cd72d8986e42/dns-node-resolver/0.log"
Sep 10 20:58:48.230290 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:48.230261    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ccgh9_27105194-1643-4b6a-ae7b-2e0d76967bb6/node-ca/0.log"
Sep 10 20:58:48.356121 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:48.356071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ctgwk\" (UID: \"91ba5603-16c5-4fa8-b573-b57774494893\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk"
Sep 10 20:58:48.356334 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:48.356293    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 20:58:48.356399 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:48.356356    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls podName:91ba5603-16c5-4fa8-b573-b57774494893 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:56.356340193 +0000 UTC m=+77.332000218 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-ctgwk" (UID: "91ba5603-16c5-4fa8-b573-b57774494893") : secret "samples-operator-tls" not found
Sep 10 20:58:49.162805 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:49.162766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:49.162996 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:49.162889    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:58:49.162996 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:49.162963    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm podName:e7fd3843-b239-4f87-ab91-808f4a31f064 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:53.162944132 +0000 UTC m=+74.138604172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dph2p" (UID: "e7fd3843-b239-4f87-ab91-808f4a31f064") : configmap "kube-rbac-proxy" not found
Sep 10 20:58:49.162996 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:49.162979    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:49.162996 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:49.162891    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:49.163223 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:49.163032    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls podName:e7fd3843-b239-4f87-ab91-808f4a31f064 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:53.163016835 +0000 UTC m=+74.138676870 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dph2p" (UID: "e7fd3843-b239-4f87-ab91-808f4a31f064") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:49.230295 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:49.230266    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c5cks_6d565bb5-5969-45f9-b862-58ccf6248c9a/serve-healthcheck-canary/0.log"
Sep 10 20:58:50.927261 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:50.927226    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lpwwj" event={"ID":"97ae132c-459a-4cba-bb9f-822279e84225","Type":"ContainerStarted","Data":"4ba4abb04b135e75a3e38afaeedaaee7f340d3077c5abbf45e10a070424ef350"}
Sep 10 20:58:50.927708 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:50.927333    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:58:50.928701 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:50.928679    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-29b5p" event={"ID":"8cefd1c8-a812-4cde-8fda-cc35af6e9833","Type":"ContainerStarted","Data":"7f18d4f2fb24fc043d413c51ac8bf2aa9dbeef78e8f4e27325f4b28c7beaf37d"}
Sep 10 20:58:50.949293 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:50.949256    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lpwwj" podStartSLOduration=66.32960252 podStartE2EDuration="1m10.949243803s" totalImagesPullingTime="4.619641283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:50.948988262 +0000 UTC m=+71.924648320" watchObservedRunningTime="2026-09-10 20:58:50.949243803 +0000 UTC m=+71.924903846"
Sep 10 20:58:50.970275 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:50.970234    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-29b5p" podStartSLOduration=65.30591422 podStartE2EDuration="1m9.970222673s" totalImagesPullingTime="4.664308453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:50.96950984 +0000 UTC m=+71.945169888" watchObservedRunningTime="2026-09-10 20:58:50.970222673 +0000 UTC m=+71.945882711"
Sep 10 20:58:53.190942 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.190908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:53.191394 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.190985    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:58:53.191394 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:53.191070    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:53.191394 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:53.191092    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:58:53.191394 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:53.191136    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls podName:e7fd3843-b239-4f87-ab91-808f4a31f064 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:01.191120284 +0000 UTC m=+82.166780309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dph2p" (UID: "e7fd3843-b239-4f87-ab91-808f4a31f064") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:53.191394 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:53.191151    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm podName:e7fd3843-b239-4f87-ab91-808f4a31f064 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:01.191144031 +0000 UTC m=+82.166804056 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dph2p" (UID: "e7fd3843-b239-4f87-ab91-808f4a31f064") : configmap "kube-rbac-proxy" not found
Sep 10 20:58:53.827298 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.827265    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-bxhqq"]
Sep 10 20:58:53.830277 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.830257    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-bxhqq"
Sep 10 20:58:53.834283 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.834264    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 10 20:58:53.834652 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.834634    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 10 20:58:53.834734 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.834638    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nkhgw\""
Sep 10 20:58:53.834907 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.834892    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 10 20:58:53.834982 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.834919    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 10 20:58:53.840784 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.840764    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-bxhqq"]
Sep 10 20:58:53.894986 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.894950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e6286393-5a40-4e6e-a5c0-8b7864369c7b-signing-key\") pod \"service-ca-568869fdc5-bxhqq\" (UID: \"e6286393-5a40-4e6e-a5c0-8b7864369c7b\") " pod="openshift-service-ca/service-ca-568869fdc5-bxhqq"
Sep 10 20:58:53.895148 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.895005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e6286393-5a40-4e6e-a5c0-8b7864369c7b-signing-cabundle\") pod \"service-ca-568869fdc5-bxhqq\" (UID: \"e6286393-5a40-4e6e-a5c0-8b7864369c7b\") " pod="openshift-service-ca/service-ca-568869fdc5-bxhqq"
Sep 10 20:58:53.895148 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.895057    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvdk9\" (UniqueName: \"kubernetes.io/projected/e6286393-5a40-4e6e-a5c0-8b7864369c7b-kube-api-access-nvdk9\") pod \"service-ca-568869fdc5-bxhqq\" (UID: \"e6286393-5a40-4e6e-a5c0-8b7864369c7b\") " pod="openshift-service-ca/service-ca-568869fdc5-bxhqq"
Sep 10 20:58:53.996216 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.996167    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvdk9\" (UniqueName: \"kubernetes.io/projected/e6286393-5a40-4e6e-a5c0-8b7864369c7b-kube-api-access-nvdk9\") pod \"service-ca-568869fdc5-bxhqq\" (UID: \"e6286393-5a40-4e6e-a5c0-8b7864369c7b\") " pod="openshift-service-ca/service-ca-568869fdc5-bxhqq"
Sep 10 20:58:53.996371 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.996264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e6286393-5a40-4e6e-a5c0-8b7864369c7b-signing-key\") pod \"service-ca-568869fdc5-bxhqq\" (UID: \"e6286393-5a40-4e6e-a5c0-8b7864369c7b\") " pod="openshift-service-ca/service-ca-568869fdc5-bxhqq"
Sep 10 20:58:53.996371 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.996302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e6286393-5a40-4e6e-a5c0-8b7864369c7b-signing-cabundle\") pod \"service-ca-568869fdc5-bxhqq\" (UID: \"e6286393-5a40-4e6e-a5c0-8b7864369c7b\") " pod="openshift-service-ca/service-ca-568869fdc5-bxhqq"
Sep 10 20:58:53.996921 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.996895    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e6286393-5a40-4e6e-a5c0-8b7864369c7b-signing-cabundle\") pod \"service-ca-568869fdc5-bxhqq\" (UID: \"e6286393-5a40-4e6e-a5c0-8b7864369c7b\") " pod="openshift-service-ca/service-ca-568869fdc5-bxhqq"
Sep 10 20:58:53.998642 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:53.998614    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e6286393-5a40-4e6e-a5c0-8b7864369c7b-signing-key\") pod \"service-ca-568869fdc5-bxhqq\" (UID: \"e6286393-5a40-4e6e-a5c0-8b7864369c7b\") " pod="openshift-service-ca/service-ca-568869fdc5-bxhqq"
Sep 10 20:58:54.005843 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:54.005821    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvdk9\" (UniqueName: \"kubernetes.io/projected/e6286393-5a40-4e6e-a5c0-8b7864369c7b-kube-api-access-nvdk9\") pod \"service-ca-568869fdc5-bxhqq\" (UID: \"e6286393-5a40-4e6e-a5c0-8b7864369c7b\") " pod="openshift-service-ca/service-ca-568869fdc5-bxhqq"
Sep 10 20:58:54.138393 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:54.138317    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-bxhqq"
Sep 10 20:58:54.407237 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:54.407157    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-bxhqq"]
Sep 10 20:58:54.410826 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:58:54.410805    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6286393_5a40_4e6e_a5c0_8b7864369c7b.slice/crio-aa89fa8ae9e04baff874341daf8597e4ca3b965c3b58149ab36ac34e1aa28382 WatchSource:0}: Error finding container aa89fa8ae9e04baff874341daf8597e4ca3b965c3b58149ab36ac34e1aa28382: Status 404 returned error can't find the container with id aa89fa8ae9e04baff874341daf8597e4ca3b965c3b58149ab36ac34e1aa28382
Sep 10 20:58:54.939409 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:54.939376    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-bxhqq" event={"ID":"e6286393-5a40-4e6e-a5c0-8b7864369c7b","Type":"ContainerStarted","Data":"aa89fa8ae9e04baff874341daf8597e4ca3b965c3b58149ab36ac34e1aa28382"}
Sep 10 20:58:56.416932 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:56.416838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ctgwk\" (UID: \"91ba5603-16c5-4fa8-b573-b57774494893\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk"
Sep 10 20:58:56.417324 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:56.416987    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 20:58:56.417324 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:58:56.417061    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls podName:91ba5603-16c5-4fa8-b573-b57774494893 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:12.41704592 +0000 UTC m=+93.392705950 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-ctgwk" (UID: "91ba5603-16c5-4fa8-b573-b57774494893") : secret "samples-operator-tls" not found
Sep 10 20:58:56.948134 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:56.948097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-bxhqq" event={"ID":"e6286393-5a40-4e6e-a5c0-8b7864369c7b","Type":"ContainerStarted","Data":"4d67316e0b1d79b9faf23b153e1d5555f3ddbff28c6bb550ab313d7dd6a13265"}
Sep 10 20:58:56.967689 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:58:56.967641    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-bxhqq" podStartSLOduration=2.282993437 podStartE2EDuration="3.967626428s" totalImagesPullingTime="1.684632991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:56.967289292 +0000 UTC m=+77.942949330" watchObservedRunningTime="2026-09-10 20:58:56.967626428 +0000 UTC m=+77.943286477"
Sep 10 20:59:01.252954 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:01.252923    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:59:01.253365 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:01.252991    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:59:01.253365 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:59:01.253093    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:01.253365 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:59:01.253142    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls podName:e7fd3843-b239-4f87-ab91-808f4a31f064 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:17.253129563 +0000 UTC m=+98.228789588 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dph2p" (UID: "e7fd3843-b239-4f87-ab91-808f4a31f064") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:01.253487 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:01.253457    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7fd3843-b239-4f87-ab91-808f4a31f064-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:59:12.434190 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:12.434152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ctgwk\" (UID: \"91ba5603-16c5-4fa8-b573-b57774494893\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk"
Sep 10 20:59:12.436520 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:12.436502    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91ba5603-16c5-4fa8-b573-b57774494893-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ctgwk\" (UID: \"91ba5603-16c5-4fa8-b573-b57774494893\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk"
Sep 10 20:59:12.736396 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:12.736298    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk"
Sep 10 20:59:13.010696 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:13.010614    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk"]
Sep 10 20:59:13.991542 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:13.991506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk" event={"ID":"91ba5603-16c5-4fa8-b573-b57774494893","Type":"ContainerStarted","Data":"eb3bb93fb519de3ba5a84997b2a4df73ec2f70a1e897807041ffc566b5bee5e0"}
Sep 10 20:59:14.995850 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:14.995818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk" event={"ID":"91ba5603-16c5-4fa8-b573-b57774494893","Type":"ContainerStarted","Data":"17b59f34dd2c4a6764153de3b2e4551c66eb915970c0b068c4aed528736f970a"}
Sep 10 20:59:14.996219 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:14.995896    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk" event={"ID":"91ba5603-16c5-4fa8-b573-b57774494893","Type":"ContainerStarted","Data":"bdd673285d7a614e28365bc67f709a15b26599ba8ea996a9bad436742bdc22ee"}
Sep 10 20:59:15.016198 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.016152    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ctgwk" podStartSLOduration=33.585481492 podStartE2EDuration="35.016120498s" totalImagesPullingTime="1.430639006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:15.016060014 +0000 UTC m=+95.991720060" watchObservedRunningTime="2026-09-10 20:59:15.016120498 +0000 UTC m=+95.991780548"
Sep 10 20:59:15.805323 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.805289    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7586c8586d-md44c"]
Sep 10 20:59:15.808242 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.808228    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7586c8586d-md44c"
Sep 10 20:59:15.811706 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.811686    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 20:59:15.812380 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.812364    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-h88jh\""
Sep 10 20:59:15.812441 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.812376    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 20:59:15.812441 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.812385    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 20:59:15.812441 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.812395    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 20:59:15.829366 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.829349    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7586c8586d-md44c"]
Sep 10 20:59:15.859477 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.859454    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02ac36be-ef88-4033-aa54-3cbae6e072f0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7586c8586d-md44c\" (UID: \"02ac36be-ef88-4033-aa54-3cbae6e072f0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7586c8586d-md44c"
Sep 10 20:59:15.859568 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.859483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drrnj\" (UniqueName: \"kubernetes.io/projected/02ac36be-ef88-4033-aa54-3cbae6e072f0-kube-api-access-drrnj\") pod \"managed-serviceaccount-addon-agent-7586c8586d-md44c\" (UID: \"02ac36be-ef88-4033-aa54-3cbae6e072f0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7586c8586d-md44c"
Sep 10 20:59:15.901346 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.901320    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-vhxrv"]
Sep 10 20:59:15.904157 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.904143    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-vhxrv"
Sep 10 20:59:15.907761 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.907744    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 20:59:15.907855 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.907786    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 20:59:15.914117 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.914103    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lp6ht\""
Sep 10 20:59:15.944038 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.944017    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-vhxrv"]
Sep 10 20:59:15.960530 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.960508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02ac36be-ef88-4033-aa54-3cbae6e072f0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7586c8586d-md44c\" (UID: \"02ac36be-ef88-4033-aa54-3cbae6e072f0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7586c8586d-md44c"
Sep 10 20:59:15.960644 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.960542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrnj\" (UniqueName: \"kubernetes.io/projected/02ac36be-ef88-4033-aa54-3cbae6e072f0-kube-api-access-drrnj\") pod \"managed-serviceaccount-addon-agent-7586c8586d-md44c\" (UID: \"02ac36be-ef88-4033-aa54-3cbae6e072f0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7586c8586d-md44c"
Sep 10 20:59:15.960644 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.960581    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz7nk\" (UniqueName: \"kubernetes.io/projected/182f6779-fcd7-40da-9002-04a208edf5a4-kube-api-access-fz7nk\") pod \"downloads-76f99584f4-vhxrv\" (UID: \"182f6779-fcd7-40da-9002-04a208edf5a4\") " pod="openshift-console/downloads-76f99584f4-vhxrv"
Sep 10 20:59:15.962915 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.962899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02ac36be-ef88-4033-aa54-3cbae6e072f0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7586c8586d-md44c\" (UID: \"02ac36be-ef88-4033-aa54-3cbae6e072f0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7586c8586d-md44c"
Sep 10 20:59:15.990033 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:15.990013    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drrnj\" (UniqueName: \"kubernetes.io/projected/02ac36be-ef88-4033-aa54-3cbae6e072f0-kube-api-access-drrnj\") pod \"managed-serviceaccount-addon-agent-7586c8586d-md44c\" (UID: \"02ac36be-ef88-4033-aa54-3cbae6e072f0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7586c8586d-md44c"
Sep 10 20:59:16.061539 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:16.061452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz7nk\" (UniqueName: \"kubernetes.io/projected/182f6779-fcd7-40da-9002-04a208edf5a4-kube-api-access-fz7nk\") pod \"downloads-76f99584f4-vhxrv\" (UID: \"182f6779-fcd7-40da-9002-04a208edf5a4\") " pod="openshift-console/downloads-76f99584f4-vhxrv"
Sep 10 20:59:16.079492 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:16.079466    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz7nk\" (UniqueName: \"kubernetes.io/projected/182f6779-fcd7-40da-9002-04a208edf5a4-kube-api-access-fz7nk\") pod \"downloads-76f99584f4-vhxrv\" (UID: \"182f6779-fcd7-40da-9002-04a208edf5a4\") " pod="openshift-console/downloads-76f99584f4-vhxrv"
Sep 10 20:59:16.129248 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:16.129224    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7586c8586d-md44c"
Sep 10 20:59:16.212637 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:16.212604    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-vhxrv"
Sep 10 20:59:16.397295 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:16.397230    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7586c8586d-md44c"]
Sep 10 20:59:16.399903 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:59:16.399877    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02ac36be_ef88_4033_aa54_3cbae6e072f0.slice/crio-0d5f4663fd5d20fb5606a82fb17de3f5b547a01dc9ec1b87108210992f63a279 WatchSource:0}: Error finding container 0d5f4663fd5d20fb5606a82fb17de3f5b547a01dc9ec1b87108210992f63a279: Status 404 returned error can't find the container with id 0d5f4663fd5d20fb5606a82fb17de3f5b547a01dc9ec1b87108210992f63a279
Sep 10 20:59:16.484991 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:16.484969    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-vhxrv"]
Sep 10 20:59:16.486117 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:59:16.486094    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182f6779_fcd7_40da_9002_04a208edf5a4.slice/crio-1b76404bb6bbfa89c6d96bd7a75f01614f863e757e00c5b2d80e983c1fc46862 WatchSource:0}: Error finding container 1b76404bb6bbfa89c6d96bd7a75f01614f863e757e00c5b2d80e983c1fc46862: Status 404 returned error can't find the container with id 1b76404bb6bbfa89c6d96bd7a75f01614f863e757e00c5b2d80e983c1fc46862
Sep 10 20:59:16.772495 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:16.772427    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gfx2n"]
Sep 10 20:59:16.776453 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:16.776437    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gfx2n"
Sep 10 20:59:16.778774 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:16.778751    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bg7ck\""
Sep 10 20:59:16.778983 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:16.778969    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 10 20:59:16.787424 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:16.787405    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gfx2n"]
Sep 10 20:59:16.868075 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:16.868038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1bd0b49b-7271-4d27-8489-b4b7aa6d9746-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-gfx2n\" (UID: \"1bd0b49b-7271-4d27-8489-b4b7aa6d9746\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gfx2n"
Sep 10 20:59:16.968658 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:16.968624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1bd0b49b-7271-4d27-8489-b4b7aa6d9746-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-gfx2n\" (UID: \"1bd0b49b-7271-4d27-8489-b4b7aa6d9746\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gfx2n"
Sep 10 20:59:16.971520 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:16.971493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1bd0b49b-7271-4d27-8489-b4b7aa6d9746-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-gfx2n\" (UID: \"1bd0b49b-7271-4d27-8489-b4b7aa6d9746\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gfx2n"
Sep 10 20:59:17.006687 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:17.006648    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7586c8586d-md44c" event={"ID":"02ac36be-ef88-4033-aa54-3cbae6e072f0","Type":"ContainerStarted","Data":"0d5f4663fd5d20fb5606a82fb17de3f5b547a01dc9ec1b87108210992f63a279"}
Sep 10 20:59:17.007898 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:17.007874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-vhxrv" event={"ID":"182f6779-fcd7-40da-9002-04a208edf5a4","Type":"ContainerStarted","Data":"1b76404bb6bbfa89c6d96bd7a75f01614f863e757e00c5b2d80e983c1fc46862"}
Sep 10 20:59:17.085573 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:17.085540    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gfx2n"
Sep 10 20:59:17.271596 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:17.271561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:59:17.273991 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:17.273966    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7fd3843-b239-4f87-ab91-808f4a31f064-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dph2p\" (UID: \"e7fd3843-b239-4f87-ab91-808f4a31f064\") " pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:59:17.316049 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:17.316013    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dph2p"
Sep 10 20:59:17.370872 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:17.370748    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gfx2n"]
Sep 10 20:59:17.376177 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:59:17.376136    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd0b49b_7271_4d27_8489_b4b7aa6d9746.slice/crio-5ceea7436f901cbe69eca7693f348b5d875f2e128e919e52eb6dbae4e872f42b WatchSource:0}: Error finding container 5ceea7436f901cbe69eca7693f348b5d875f2e128e919e52eb6dbae4e872f42b: Status 404 returned error can't find the container with id 5ceea7436f901cbe69eca7693f348b5d875f2e128e919e52eb6dbae4e872f42b
Sep 10 20:59:17.609073 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:17.609032    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dph2p"]
Sep 10 20:59:17.616016 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:59:17.615985    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7fd3843_b239_4f87_ab91_808f4a31f064.slice/crio-8adabe6134350c2d8ecfdf32ea986f4bd99f502db2acfde82aef8cba945b580d WatchSource:0}: Error finding container 8adabe6134350c2d8ecfdf32ea986f4bd99f502db2acfde82aef8cba945b580d: Status 404 returned error can't find the container with id 8adabe6134350c2d8ecfdf32ea986f4bd99f502db2acfde82aef8cba945b580d
Sep 10 20:59:17.775622 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:17.775585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:59:17.775804 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:17.775654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:59:17.778749 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:17.778698    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c25e03f-53c0-43c0-ad21-7301f930243f-metrics-tls\") pod \"dns-default-7mplf\" (UID: \"9c25e03f-53c0-43c0-ad21-7301f930243f\") " pod="openshift-dns/dns-default-7mplf"
Sep 10 20:59:17.778866 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:17.778849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls\") pod \"image-registry-f469fbc67-26fpg\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") " pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:59:17.781788 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:17.781763    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8bbrk\""
Sep 10 20:59:17.789839 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:17.789813    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:59:17.814275 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:17.814245    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9hkk2\""
Sep 10 20:59:17.821994 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:17.821970    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7mplf"
Sep 10 20:59:18.020191 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.020154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dph2p" event={"ID":"e7fd3843-b239-4f87-ab91-808f4a31f064","Type":"ContainerStarted","Data":"8e1bbb900f3ab822c0ecaefc5ff2ab7c2d5d7ee10fb08f4a35d390bb782325c8"}
Sep 10 20:59:18.020383 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.020247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dph2p" event={"ID":"e7fd3843-b239-4f87-ab91-808f4a31f064","Type":"ContainerStarted","Data":"8adabe6134350c2d8ecfdf32ea986f4bd99f502db2acfde82aef8cba945b580d"}
Sep 10 20:59:18.022024 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.021984    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gfx2n" event={"ID":"1bd0b49b-7271-4d27-8489-b4b7aa6d9746","Type":"ContainerStarted","Data":"5ceea7436f901cbe69eca7693f348b5d875f2e128e919e52eb6dbae4e872f42b"}
Sep 10 20:59:18.106631 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.106598    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f469fbc67-26fpg"]
Sep 10 20:59:18.115261 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:59:18.115232    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd68b468_3b12_4738_b092_1f3e6e3c3356.slice/crio-703262f5a1cdb719618852d45bb23debe92261231f42b90dca321f564ae5e80e WatchSource:0}: Error finding container 703262f5a1cdb719618852d45bb23debe92261231f42b90dca321f564ae5e80e: Status 404 returned error can't find the container with id 703262f5a1cdb719618852d45bb23debe92261231f42b90dca321f564ae5e80e
Sep 10 20:59:18.154518 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.154493    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7mplf"]
Sep 10 20:59:18.160166 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:59:18.160145    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c25e03f_53c0_43c0_ad21_7301f930243f.slice/crio-89fafb8664e59c16665bcce4c596016390995ffa8e2320844f1e783bc8a83fde WatchSource:0}: Error finding container 89fafb8664e59c16665bcce4c596016390995ffa8e2320844f1e783bc8a83fde: Status 404 returned error can't find the container with id 89fafb8664e59c16665bcce4c596016390995ffa8e2320844f1e783bc8a83fde
Sep 10 20:59:18.391224 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.391131    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5946d7ffd9-5mhnj"]
Sep 10 20:59:18.394480 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.394448    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.398026 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.398007    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 20:59:18.398257 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.398235    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 20:59:18.398486 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.398469    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 20:59:18.401174 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.401145    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-66nzk\""
Sep 10 20:59:18.401284 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.401231    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 20:59:18.401490 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.401451    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 20:59:18.407097 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.407041    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5946d7ffd9-5mhnj"]
Sep 10 20:59:18.482694 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.482657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-oauth-serving-cert\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.482857 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.482714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-oauth-config\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.482857 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.482741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hztm2\" (UniqueName: \"kubernetes.io/projected/7be70674-f7fc-4d4f-a712-c61c7e651c3a-kube-api-access-hztm2\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.482857 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.482827    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-service-ca\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.483023 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.482872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-serving-cert\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.483023 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.482901    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-config\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.584099 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.584060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-oauth-serving-cert\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.584287 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.584113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-oauth-config\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.584287 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.584136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hztm2\" (UniqueName: \"kubernetes.io/projected/7be70674-f7fc-4d4f-a712-c61c7e651c3a-kube-api-access-hztm2\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.584287 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.584166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-service-ca\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.584287 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.584189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-serving-cert\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.584287 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.584223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-config\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.585535 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.584911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-oauth-serving-cert\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.585535 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.584981    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-service-ca\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.585535 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.585491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-config\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.587525 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.587477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-oauth-config\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.588017 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.587997    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-serving-cert\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.594808 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.594785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hztm2\" (UniqueName: \"kubernetes.io/projected/7be70674-f7fc-4d4f-a712-c61c7e651c3a-kube-api-access-hztm2\") pod \"console-5946d7ffd9-5mhnj\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") " pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:18.708346 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:18.708268    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:19.031054 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:19.030851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f469fbc67-26fpg" event={"ID":"dd68b468-3b12-4738-b092-1f3e6e3c3356","Type":"ContainerStarted","Data":"8a9c99d1cb5babcc4a4fd6cf25dc7bac2ba2dfd94c835be775e5686a1f1216c4"}
Sep 10 20:59:19.031054 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:19.030902    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f469fbc67-26fpg" event={"ID":"dd68b468-3b12-4738-b092-1f3e6e3c3356","Type":"ContainerStarted","Data":"703262f5a1cdb719618852d45bb23debe92261231f42b90dca321f564ae5e80e"}
Sep 10 20:59:19.031054 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:19.030946    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:59:19.033066 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:19.033036    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7mplf" event={"ID":"9c25e03f-53c0-43c0-ad21-7301f930243f","Type":"ContainerStarted","Data":"89fafb8664e59c16665bcce4c596016390995ffa8e2320844f1e783bc8a83fde"}
Sep 10 20:59:19.056731 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:19.056679    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f469fbc67-26fpg" podStartSLOduration=98.056658207 podStartE2EDuration="1m38.056658207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:19.055012455 +0000 UTC m=+100.030672512" watchObservedRunningTime="2026-09-10 20:59:19.056658207 +0000 UTC m=+100.032318256"
Sep 10 20:59:20.287328 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.287290    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5946d7ffd9-5mhnj"]
Sep 10 20:59:20.558267 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.558169    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-565f5f457b-phlc5"]
Sep 10 20:59:20.562680 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.562640    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.570379 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.570351    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 20:59:20.574877 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.574853    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-565f5f457b-phlc5"]
Sep 10 20:59:20.602777 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:59:20.602750    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be70674_f7fc_4d4f_a712_c61c7e651c3a.slice/crio-e03c775cf5b5ef9c687df10fe0b5700ccb6f50f635f621ac23355962f1647855 WatchSource:0}: Error finding container e03c775cf5b5ef9c687df10fe0b5700ccb6f50f635f621ac23355962f1647855: Status 404 returned error can't find the container with id e03c775cf5b5ef9c687df10fe0b5700ccb6f50f635f621ac23355962f1647855
Sep 10 20:59:20.702839 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.702806    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89dded4f-6ba3-47b6-9134-307795dee576-console-serving-cert\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.702926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.702885    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89dded4f-6ba3-47b6-9134-307795dee576-console-oauth-config\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.702926 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.702916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p52g\" (UniqueName: \"kubernetes.io/projected/89dded4f-6ba3-47b6-9134-307795dee576-kube-api-access-8p52g\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.703043 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.702943    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-oauth-serving-cert\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.703043 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.702973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-service-ca\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.703043 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.703006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-trusted-ca-bundle\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.703043 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.703036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-console-config\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.803443 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.803409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p52g\" (UniqueName: \"kubernetes.io/projected/89dded4f-6ba3-47b6-9134-307795dee576-kube-api-access-8p52g\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.803542 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.803462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-oauth-serving-cert\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.803542 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.803501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-service-ca\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.803542 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.803533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-trusted-ca-bundle\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.803702 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.803567    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-console-config\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.803702 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.803636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89dded4f-6ba3-47b6-9134-307795dee576-console-serving-cert\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.803702 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.803685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89dded4f-6ba3-47b6-9134-307795dee576-console-oauth-config\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.805123 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.804987    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-trusted-ca-bundle\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.805868 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.805843    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-oauth-serving-cert\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.805988 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.805964    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-console-config\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.806055 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.805973    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-service-ca\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.807448 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.807425    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89dded4f-6ba3-47b6-9134-307795dee576-console-oauth-config\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.808658 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.808598    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89dded4f-6ba3-47b6-9134-307795dee576-console-serving-cert\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.811749 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.811724    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p52g\" (UniqueName: \"kubernetes.io/projected/89dded4f-6ba3-47b6-9134-307795dee576-kube-api-access-8p52g\") pod \"console-565f5f457b-phlc5\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") " pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:20.875151 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:20.875123    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:21.042678 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:21.042632    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7586c8586d-md44c" event={"ID":"02ac36be-ef88-4033-aa54-3cbae6e072f0","Type":"ContainerStarted","Data":"302680b87a9fc29d4d7287320429dd9318e9dda480e44577806616ad7377a23b"}
Sep 10 20:59:21.044927 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:21.044894    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dph2p" event={"ID":"e7fd3843-b239-4f87-ab91-808f4a31f064","Type":"ContainerStarted","Data":"405ab7e1ad86d8e7827494c82b928aa9dca857803dd72716ae6009856136b561"}
Sep 10 20:59:21.047340 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:21.047314    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gfx2n" event={"ID":"1bd0b49b-7271-4d27-8489-b4b7aa6d9746","Type":"ContainerStarted","Data":"16caa6be1369280d97b9682a85a174b19b2ebc4a6006eba093a296859eb4f5d2"}
Sep 10 20:59:21.047992 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:21.047787    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gfx2n"
Sep 10 20:59:21.049902 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:21.049879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7mplf" event={"ID":"9c25e03f-53c0-43c0-ad21-7301f930243f","Type":"ContainerStarted","Data":"c816eb1620b41eb77069360efde1ded71b36515085144676266cb0879202987f"}
Sep 10 20:59:21.049999 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:21.049924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7mplf" event={"ID":"9c25e03f-53c0-43c0-ad21-7301f930243f","Type":"ContainerStarted","Data":"9c20e13f29f081dd3a123d5b7664a5a9bbad61e37c20bf428965b1fa818e0bc5"}
Sep 10 20:59:21.050055 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:21.050046    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7mplf"
Sep 10 20:59:21.052196 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:21.052175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5946d7ffd9-5mhnj" event={"ID":"7be70674-f7fc-4d4f-a712-c61c7e651c3a","Type":"ContainerStarted","Data":"e03c775cf5b5ef9c687df10fe0b5700ccb6f50f635f621ac23355962f1647855"}
Sep 10 20:59:21.054000 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:21.053979    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gfx2n"
Sep 10 20:59:21.068366 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:21.068140    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7586c8586d-md44c" podStartSLOduration=1.8569699210000001 podStartE2EDuration="6.068121571s" totalImagesPullingTime="4.21115165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:21.067349145 +0000 UTC m=+102.043009193" watchObservedRunningTime="2026-09-10 20:59:21.068121571 +0000 UTC m=+102.043781620"
Sep 10 20:59:21.087482 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:21.087438    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gfx2n" podStartSLOduration=2.450613218 podStartE2EDuration="5.087419925s" totalImagesPullingTime="2.636806707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:21.084598999 +0000 UTC m=+102.060259047" watchObservedRunningTime="2026-09-10 20:59:21.087419925 +0000 UTC m=+102.063079973"
Sep 10 20:59:21.112167 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:21.112095    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7mplf" podStartSLOduration=65.615299106 podStartE2EDuration="1m8.112061936s" totalImagesPullingTime="2.49676283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:21.111758031 +0000 UTC m=+102.087418080" watchObservedRunningTime="2026-09-10 20:59:21.112061936 +0000 UTC m=+102.087721983"
Sep 10 20:59:21.167179 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:21.167146    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-565f5f457b-phlc5"]
Sep 10 20:59:21.934970 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:21.934161    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lpwwj"
Sep 10 20:59:22.060139 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:22.060074    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565f5f457b-phlc5" event={"ID":"89dded4f-6ba3-47b6-9134-307795dee576","Type":"ContainerStarted","Data":"be69c81ade14f7ab1092a6bab69a3b3348f97a0d88d27285855ddd0dfcca5d24"}
Sep 10 20:59:23.068577 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:23.068537    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dph2p" event={"ID":"e7fd3843-b239-4f87-ab91-808f4a31f064","Type":"ContainerStarted","Data":"95a9e77161514c8032f5c94f16cef5ce2248478a2c2df6fdf74431868e86f36a"}
Sep 10 20:59:23.091605 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:23.091550    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dph2p" podStartSLOduration=33.710881447 podStartE2EDuration="38.091528122s" totalImagesPullingTime="4.380646675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:23.09057068 +0000 UTC m=+104.066230728" watchObservedRunningTime="2026-09-10 20:59:23.091528122 +0000 UTC m=+104.067188170"
Sep 10 20:59:25.078344 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:25.078306    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5946d7ffd9-5mhnj" event={"ID":"7be70674-f7fc-4d4f-a712-c61c7e651c3a","Type":"ContainerStarted","Data":"dd777f83413b78d4439cdb8deecf4aad4b073be2d9227e218a28289ffcb2c7dd"}
Sep 10 20:59:25.080288 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:25.080260    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565f5f457b-phlc5" event={"ID":"89dded4f-6ba3-47b6-9134-307795dee576","Type":"ContainerStarted","Data":"1199bb4496070c7602ad744528b6723b6a897658f4f4efbfcb4c27220d9e0d31"}
Sep 10 20:59:25.118328 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:25.118181    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5946d7ffd9-5mhnj" podStartSLOduration=3.168366883 podStartE2EDuration="7.118163616s" totalImagesPullingTime="3.949796733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:25.099432541 +0000 UTC m=+106.075092589" watchObservedRunningTime="2026-09-10 20:59:25.118163616 +0000 UTC m=+106.093823663"
Sep 10 20:59:25.118483 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:25.118407    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-565f5f457b-phlc5" podStartSLOduration=1.7295982730000001 podStartE2EDuration="5.118395036s" totalImagesPullingTime="3.388796763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:25.117821034 +0000 UTC m=+106.093481085" watchObservedRunningTime="2026-09-10 20:59:25.118395036 +0000 UTC m=+106.094055087"
Sep 10 20:59:26.275743 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.274387    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"]
Sep 10 20:59:26.279220 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.279180    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.284949 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.284926    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 20:59:26.289255 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.288831    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 20:59:26.289255 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.289038    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 10 20:59:26.290056 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.290001    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bnmbw\""
Sep 10 20:59:26.290056 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.290035    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 20:59:26.291662 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.291496    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 10 20:59:26.298760 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.298742    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 10 20:59:26.308371 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.307010    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s5sx4"]
Sep 10 20:59:26.310557 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.310540    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.321395 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.314459    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 20:59:26.321395 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.314706    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 20:59:26.321395 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.314929    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jp262\""
Sep 10 20:59:26.321395 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.315099    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 20:59:26.321395 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.316446    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"]
Sep 10 20:59:26.352435 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.352402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.352580 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.352462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.352580 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.352520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.352689 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.352580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.352689 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.352647    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6wz7\" (UniqueName: \"kubernetes.io/projected/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-api-access-d6wz7\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.352794 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.352683    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.454219 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.454108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6wz7\" (UniqueName: \"kubernetes.io/projected/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-api-access-d6wz7\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.454219 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.454159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.454219 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.454193    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/015b0f46-2fc7-4c70-8200-f95bc441ac76-node-exporter-accelerators-collector-config\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.454498 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.454249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/015b0f46-2fc7-4c70-8200-f95bc441ac76-node-exporter-tls\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.454498 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.454287    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.454498 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.454310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/015b0f46-2fc7-4c70-8200-f95bc441ac76-sys\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.454498 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.454357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.454498 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.454389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/015b0f46-2fc7-4c70-8200-f95bc441ac76-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.454498 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.454420    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/015b0f46-2fc7-4c70-8200-f95bc441ac76-root\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.454498 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.454442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/015b0f46-2fc7-4c70-8200-f95bc441ac76-metrics-client-ca\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.454498 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.454468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krsp6\" (UniqueName: \"kubernetes.io/projected/015b0f46-2fc7-4c70-8200-f95bc441ac76-kube-api-access-krsp6\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.454898 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.454505    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/015b0f46-2fc7-4c70-8200-f95bc441ac76-node-exporter-wtmp\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.454898 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.454544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.454898 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.454580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.454898 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.454613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/015b0f46-2fc7-4c70-8200-f95bc441ac76-node-exporter-textfile\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.458216 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.455678    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.458216 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.455982    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.458216 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:59:26.456465    2581 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 10 20:59:26.458216 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:59:26.456537    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-state-metrics-tls podName:ce584ac9-fc93-43da-8cb2-2f7082a2b59a nodeName:}" failed. No retries permitted until 2026-09-10 20:59:26.956517167 +0000 UTC m=+107.932177205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-vqvpj" (UID: "ce584ac9-fc93-43da-8cb2-2f7082a2b59a") : secret "kube-state-metrics-tls" not found
Sep 10 20:59:26.458216 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.456857    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.475671 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.472630    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.478791 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.478749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6wz7\" (UniqueName: \"kubernetes.io/projected/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-api-access-d6wz7\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.555480 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.555405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/015b0f46-2fc7-4c70-8200-f95bc441ac76-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.555480 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.555443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/015b0f46-2fc7-4c70-8200-f95bc441ac76-root\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.555480 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.555470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/015b0f46-2fc7-4c70-8200-f95bc441ac76-metrics-client-ca\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.555724 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.555495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krsp6\" (UniqueName: \"kubernetes.io/projected/015b0f46-2fc7-4c70-8200-f95bc441ac76-kube-api-access-krsp6\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.555724 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.555526    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/015b0f46-2fc7-4c70-8200-f95bc441ac76-node-exporter-wtmp\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.555724 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.555584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/015b0f46-2fc7-4c70-8200-f95bc441ac76-node-exporter-textfile\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.555724 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.555634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/015b0f46-2fc7-4c70-8200-f95bc441ac76-node-exporter-accelerators-collector-config\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.555724 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.555666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/015b0f46-2fc7-4c70-8200-f95bc441ac76-node-exporter-tls\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.555724 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.555694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/015b0f46-2fc7-4c70-8200-f95bc441ac76-sys\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.555999 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.555797    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/015b0f46-2fc7-4c70-8200-f95bc441ac76-sys\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.555999 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.555937    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/015b0f46-2fc7-4c70-8200-f95bc441ac76-node-exporter-wtmp\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.555999 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.555967    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/015b0f46-2fc7-4c70-8200-f95bc441ac76-root\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.556356 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.556332    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/015b0f46-2fc7-4c70-8200-f95bc441ac76-node-exporter-textfile\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.556673 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.556648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/015b0f46-2fc7-4c70-8200-f95bc441ac76-node-exporter-accelerators-collector-config\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.557109 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.557084    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/015b0f46-2fc7-4c70-8200-f95bc441ac76-metrics-client-ca\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.558649 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.558623    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/015b0f46-2fc7-4c70-8200-f95bc441ac76-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.559481 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.559462    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/015b0f46-2fc7-4c70-8200-f95bc441ac76-node-exporter-tls\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.571386 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.571340    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krsp6\" (UniqueName: \"kubernetes.io/projected/015b0f46-2fc7-4c70-8200-f95bc441ac76-kube-api-access-krsp6\") pod \"node-exporter-s5sx4\" (UID: \"015b0f46-2fc7-4c70-8200-f95bc441ac76\") " pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.632130 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.632102    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s5sx4"
Sep 10 20:59:26.644790 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:59:26.644766    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015b0f46_2fc7_4c70_8200_f95bc441ac76.slice/crio-19cf91af230251a9c4aaa7618fa0f7c395437c619e6600b79075b3abd1980698 WatchSource:0}: Error finding container 19cf91af230251a9c4aaa7618fa0f7c395437c619e6600b79075b3abd1980698: Status 404 returned error can't find the container with id 19cf91af230251a9c4aaa7618fa0f7c395437c619e6600b79075b3abd1980698
Sep 10 20:59:26.960637 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:26.960555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:26.960786 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:59:26.960693    2581 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 10 20:59:26.960786 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:59:26.960755    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-state-metrics-tls podName:ce584ac9-fc93-43da-8cb2-2f7082a2b59a nodeName:}" failed. No retries permitted until 2026-09-10 20:59:27.960737129 +0000 UTC m=+108.936397159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-vqvpj" (UID: "ce584ac9-fc93-43da-8cb2-2f7082a2b59a") : secret "kube-state-metrics-tls" not found
Sep 10 20:59:27.090543 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.090501    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5sx4" event={"ID":"015b0f46-2fc7-4c70-8200-f95bc441ac76","Type":"ContainerStarted","Data":"19cf91af230251a9c4aaa7618fa0f7c395437c619e6600b79075b3abd1980698"}
Sep 10 20:59:27.365455 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.365420    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:59:27.370300 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.370280    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.373009 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.372985    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 20:59:27.373638 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.373619    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 20:59:27.373811 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.373789    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 20:59:27.374328 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.374147    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 20:59:27.374328 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.374165    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 20:59:27.374328 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.374182    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 20:59:27.374328 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.374170    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 20:59:27.374328 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.374152    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 20:59:27.374328 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.374289    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4rpfc\""
Sep 10 20:59:27.374685 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.374577    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 20:59:27.387775 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.387748    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:59:27.465067 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.465041    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.465255 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.465093    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz69f\" (UniqueName: \"kubernetes.io/projected/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-kube-api-access-cz69f\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.465255 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.465160    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.465387 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.465261    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.465387 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.465326    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.465387 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.465353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.465525 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.465397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.465525 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.465420    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-config-out\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.465525 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.465471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.465651 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.465530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.465651 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.465574    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.465651 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.465601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-web-config\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.465787 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.465658    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.566643 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.566601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.566823 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.566654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.566823 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.566700    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz69f\" (UniqueName: \"kubernetes.io/projected/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-kube-api-access-cz69f\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.566823 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.566739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.566823 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.566767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.566823 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:59:27.566779    2581 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 10 20:59:27.566823 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.566820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.567130 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:59:27.566852    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-main-tls podName:8e733ef7-73f2-40cd-bfeb-020645b2b3ee nodeName:}" failed. No retries permitted until 2026-09-10 20:59:28.066832846 +0000 UTC m=+109.042492891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "8e733ef7-73f2-40cd-bfeb-020645b2b3ee") : secret "alertmanager-main-tls" not found
Sep 10 20:59:27.567130 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.566890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.567130 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.566948    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.567130 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.566973    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-config-out\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.567130 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.567009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.567130 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.567038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.567130 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.567061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.567130 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.567085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-web-config\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.567653 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.567632    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.567768 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.567749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.567902 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.567878    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.570279 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.570236    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-config-out\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.570457 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.570436    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.570576 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.570549    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.570576 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.570570    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.571343 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.571299    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.571582 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.571558    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.572388 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.572365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-web-config\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.572943 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.572919    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.579260 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.579228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz69f\" (UniqueName: \"kubernetes.io/projected/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-kube-api-access-cz69f\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:27.971490 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.971454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:27.974277 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:27.974242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce584ac9-fc93-43da-8cb2-2f7082a2b59a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-vqvpj\" (UID: \"ce584ac9-fc93-43da-8cb2-2f7082a2b59a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:28.072438 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.072404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:28.075193 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.075166    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:28.093301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.093280    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"
Sep 10 20:59:28.281304 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.281231    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:28.322066 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.322036    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-684c958f79-l44l5"]
Sep 10 20:59:28.325707 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.325683    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.328463 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.328150    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 10 20:59:28.328463 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.328150    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 10 20:59:28.328463 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.328259    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 10 20:59:28.328463 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.328149    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 10 20:59:28.328463 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.328266    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 10 20:59:28.328736 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.328512    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-g8958\""
Sep 10 20:59:28.328736 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.328698    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-96tv51orqhksc\""
Sep 10 20:59:28.339572 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.339545    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-684c958f79-l44l5"]
Sep 10 20:59:28.475637 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.475601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-metrics-client-ca\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.476048 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.475669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.476048 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.475746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgljh\" (UniqueName: \"kubernetes.io/projected/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-kube-api-access-xgljh\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.476048 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.475816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.476048 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.475858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.476048 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.475903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-grpc-tls\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.476048 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.475956    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-thanos-querier-tls\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.476048 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.476002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.577094 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.577060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.577094 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.577098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgljh\" (UniqueName: \"kubernetes.io/projected/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-kube-api-access-xgljh\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.577370 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.577122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.577370 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.577149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.577370 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.577334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-grpc-tls\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.577518 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.577408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-thanos-querier-tls\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.577518 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.577460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.577595 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.577518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-metrics-client-ca\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.578380 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.578341    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-metrics-client-ca\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.580217 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.580176    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.580315 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.580263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.580511 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.580486    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.580782 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.580765    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-grpc-tls\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.580861 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.580802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-thanos-querier-tls\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.580861 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.580819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.586786 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.586767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgljh\" (UniqueName: \"kubernetes.io/projected/c4adbb2b-130d-4a30-beb9-d8e8c438acc9-kube-api-access-xgljh\") pod \"thanos-querier-684c958f79-l44l5\" (UID: \"c4adbb2b-130d-4a30-beb9-d8e8c438acc9\") " pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.637378 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.637353    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:28.708766 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.708729    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:28.708917 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.708806    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:28.714346 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:28.714322    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:29.102667 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:29.102638    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:30.534414 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.534376    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"]
Sep 10 20:59:30.537896 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.537858    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.540910 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.540877    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 10 20:59:30.541059 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.540928    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 10 20:59:30.541059 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.540939    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 10 20:59:30.541196 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.541168    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ntr8b\""
Sep 10 20:59:30.541477 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.541457    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dqa9tqces49j3\""
Sep 10 20:59:30.541570 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.541494    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 20:59:30.547986 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.547965    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"]
Sep 10 20:59:30.697702 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.697664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/188ff6df-fa09-4db5-a54c-82fb0bb440bc-secret-metrics-server-client-certs\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.697702 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.697706    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/188ff6df-fa09-4db5-a54c-82fb0bb440bc-metrics-server-audit-profiles\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.697939 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.697735    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h892b\" (UniqueName: \"kubernetes.io/projected/188ff6df-fa09-4db5-a54c-82fb0bb440bc-kube-api-access-h892b\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.697939 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.697848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/188ff6df-fa09-4db5-a54c-82fb0bb440bc-client-ca-bundle\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.697939 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.697918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/188ff6df-fa09-4db5-a54c-82fb0bb440bc-secret-metrics-server-tls\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.698111 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.697952    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/188ff6df-fa09-4db5-a54c-82fb0bb440bc-audit-log\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.698111 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.698024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/188ff6df-fa09-4db5-a54c-82fb0bb440bc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.799282 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.799157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/188ff6df-fa09-4db5-a54c-82fb0bb440bc-secret-metrics-server-client-certs\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.799282 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.799247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/188ff6df-fa09-4db5-a54c-82fb0bb440bc-metrics-server-audit-profiles\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.799282 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.799269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h892b\" (UniqueName: \"kubernetes.io/projected/188ff6df-fa09-4db5-a54c-82fb0bb440bc-kube-api-access-h892b\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.799547 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.799330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/188ff6df-fa09-4db5-a54c-82fb0bb440bc-client-ca-bundle\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.799547 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.799382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/188ff6df-fa09-4db5-a54c-82fb0bb440bc-secret-metrics-server-tls\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.799547 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.799410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/188ff6df-fa09-4db5-a54c-82fb0bb440bc-audit-log\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.799547 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.799454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/188ff6df-fa09-4db5-a54c-82fb0bb440bc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.802487 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.802433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/188ff6df-fa09-4db5-a54c-82fb0bb440bc-metrics-server-audit-profiles\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.803079 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.802782    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/188ff6df-fa09-4db5-a54c-82fb0bb440bc-client-ca-bundle\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.803079 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.803023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/188ff6df-fa09-4db5-a54c-82fb0bb440bc-audit-log\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.803079 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.803038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/188ff6df-fa09-4db5-a54c-82fb0bb440bc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.803268 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.803108    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/188ff6df-fa09-4db5-a54c-82fb0bb440bc-secret-metrics-server-client-certs\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.803268 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.803118    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/188ff6df-fa09-4db5-a54c-82fb0bb440bc-secret-metrics-server-tls\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.808044 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.808014    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h892b\" (UniqueName: \"kubernetes.io/projected/188ff6df-fa09-4db5-a54c-82fb0bb440bc-kube-api-access-h892b\") pod \"metrics-server-7bcbdfdfbf-85x6w\" (UID: \"188ff6df-fa09-4db5-a54c-82fb0bb440bc\") " pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.848978 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.848942    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:30.875615 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.875581    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:30.875716 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.875636    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:30.881098 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:30.881076    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:31.062357 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:31.062281    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7mplf"
Sep 10 20:59:31.111747 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:31.111718    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 20:59:31.181849 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:31.181816    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5946d7ffd9-5mhnj"]
Sep 10 20:59:34.246599 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:34.246504    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:59:34.252777 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:34.252261    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"]
Sep 10 20:59:34.259904 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:59:34.256291    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e733ef7_73f2_40cd_bfeb_020645b2b3ee.slice/crio-1e2320a2b40b94b172d59aa106fd0010dbed97959af3c0e893553375d8024f65 WatchSource:0}: Error finding container 1e2320a2b40b94b172d59aa106fd0010dbed97959af3c0e893553375d8024f65: Status 404 returned error can't find the container with id 1e2320a2b40b94b172d59aa106fd0010dbed97959af3c0e893553375d8024f65
Sep 10 20:59:34.259904 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:34.258530    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj"]
Sep 10 20:59:34.259904 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:59:34.259510    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188ff6df_fa09_4db5_a54c_82fb0bb440bc.slice/crio-09b8b648c86c7cc0c503ddf8f1ca1bb855196cb9ab751c748cdb1a679072ad25 WatchSource:0}: Error finding container 09b8b648c86c7cc0c503ddf8f1ca1bb855196cb9ab751c748cdb1a679072ad25: Status 404 returned error can't find the container with id 09b8b648c86c7cc0c503ddf8f1ca1bb855196cb9ab751c748cdb1a679072ad25
Sep 10 20:59:34.264669 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:59:34.264440    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce584ac9_fc93_43da_8cb2_2f7082a2b59a.slice/crio-0f663ea4c15e9680c306e1deafb1ff6c94bb8c94732374540828260e42b5a2ab WatchSource:0}: Error finding container 0f663ea4c15e9680c306e1deafb1ff6c94bb8c94732374540828260e42b5a2ab: Status 404 returned error can't find the container with id 0f663ea4c15e9680c306e1deafb1ff6c94bb8c94732374540828260e42b5a2ab
Sep 10 20:59:34.265684 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:34.265233    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-684c958f79-l44l5"]
Sep 10 20:59:34.268282 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:59:34.268257    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4adbb2b_130d_4a30_beb9_d8e8c438acc9.slice/crio-592db8289122fe6bdbb746042596116f351ff21956a2d4b30caf2530e0a454e4 WatchSource:0}: Error finding container 592db8289122fe6bdbb746042596116f351ff21956a2d4b30caf2530e0a454e4: Status 404 returned error can't find the container with id 592db8289122fe6bdbb746042596116f351ff21956a2d4b30caf2530e0a454e4
Sep 10 20:59:35.126838 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:35.126794    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-vhxrv" event={"ID":"182f6779-fcd7-40da-9002-04a208edf5a4","Type":"ContainerStarted","Data":"9e0be2d559a143e64777cac257d77c25c9a8762ca0674fb74b7a2d6a71db9cba"}
Sep 10 20:59:35.127529 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:35.127505    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-vhxrv"
Sep 10 20:59:35.133576 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:35.133538    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj" event={"ID":"ce584ac9-fc93-43da-8cb2-2f7082a2b59a","Type":"ContainerStarted","Data":"0f663ea4c15e9680c306e1deafb1ff6c94bb8c94732374540828260e42b5a2ab"}
Sep 10 20:59:35.135624 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:35.135592    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-684c958f79-l44l5" event={"ID":"c4adbb2b-130d-4a30-beb9-d8e8c438acc9","Type":"ContainerStarted","Data":"592db8289122fe6bdbb746042596116f351ff21956a2d4b30caf2530e0a454e4"}
Sep 10 20:59:35.138029 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:35.138000    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w" event={"ID":"188ff6df-fa09-4db5-a54c-82fb0bb440bc","Type":"ContainerStarted","Data":"09b8b648c86c7cc0c503ddf8f1ca1bb855196cb9ab751c748cdb1a679072ad25"}
Sep 10 20:59:35.140676 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:35.140651    2581 generic.go:356] "Generic (PLEG): container finished" podID="015b0f46-2fc7-4c70-8200-f95bc441ac76" containerID="d3ff28136cb5840c58f1f8e872050852a033f0ace28f1198a287918c8640bac7" exitCode=0
Sep 10 20:59:35.140780 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:35.140735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5sx4" event={"ID":"015b0f46-2fc7-4c70-8200-f95bc441ac76","Type":"ContainerDied","Data":"d3ff28136cb5840c58f1f8e872050852a033f0ace28f1198a287918c8640bac7"}
Sep 10 20:59:35.142487 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:35.142461    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e733ef7-73f2-40cd-bfeb-020645b2b3ee","Type":"ContainerStarted","Data":"1e2320a2b40b94b172d59aa106fd0010dbed97959af3c0e893553375d8024f65"}
Sep 10 20:59:35.146012 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:35.145990    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-vhxrv"
Sep 10 20:59:35.152960 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:35.152899    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-vhxrv" podStartSLOduration=2.577440726 podStartE2EDuration="20.152882591s" totalImagesPullingTime="17.575441865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:35.151844286 +0000 UTC m=+116.127504334" watchObservedRunningTime="2026-09-10 20:59:35.152882591 +0000 UTC m=+116.128542640"
Sep 10 20:59:36.152301 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.152264    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5sx4" event={"ID":"015b0f46-2fc7-4c70-8200-f95bc441ac76","Type":"ContainerStarted","Data":"f0afb9580e65ca43a539a22470199362e26c258142d10cba9f9094bc57401e8d"}
Sep 10 20:59:36.152852 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.152346    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5sx4" event={"ID":"015b0f46-2fc7-4c70-8200-f95bc441ac76","Type":"ContainerStarted","Data":"284cb6de0d06a1d5b6c7f66a59ab0bf1d6c50de087bf58218b69b854fabc1d6f"}
Sep 10 20:59:36.172544 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.172512    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d69556c9f-7kz25"]
Sep 10 20:59:36.190713 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.190665    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s5sx4" podStartSLOduration=2.039530747 podStartE2EDuration="10.190648187s" totalImagesPullingTime="8.058528812s" totalInitContainerRuntime="92.588628ms" isStatefulPod=true podCreationTimestamp="2026-09-10 20:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:36.176601971 +0000 UTC m=+117.152262017" watchObservedRunningTime="2026-09-10 20:59:36.190648187 +0000 UTC m=+117.166308234"
Sep 10 20:59:36.193531 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.193407    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d69556c9f-7kz25"]
Sep 10 20:59:36.193531 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.193519    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.357642 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.357568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-trusted-ca-bundle\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.358043 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.357722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-config\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.358043 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.357840    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-serving-cert\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.358043 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.357874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-service-ca\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.358043 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.357910    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-oauth-serving-cert\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.358334 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.358169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-oauth-config\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.358334 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.358230    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcqcq\" (UniqueName: \"kubernetes.io/projected/3129da09-6cf1-49cf-a775-166ef15e9f3d-kube-api-access-hcqcq\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.459375 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.459285    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-serving-cert\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.459804 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.459392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-service-ca\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.459804 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.459657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-oauth-serving-cert\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.459804 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.459698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-oauth-config\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.459804 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.459767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcqcq\" (UniqueName: \"kubernetes.io/projected/3129da09-6cf1-49cf-a775-166ef15e9f3d-kube-api-access-hcqcq\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.460032 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.459857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-trusted-ca-bundle\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.460032 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.459895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-config\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.460442 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.460323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-service-ca\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.460740 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.460595    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-config\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.461056 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.461031    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-oauth-serving-cert\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.461162 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.461127    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-trusted-ca-bundle\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.463679 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.463505    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-serving-cert\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.463789 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.463689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-oauth-config\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.469603 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.469581    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcqcq\" (UniqueName: \"kubernetes.io/projected/3129da09-6cf1-49cf-a775-166ef15e9f3d-kube-api-access-hcqcq\") pod \"console-7d69556c9f-7kz25\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") " pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:36.506627 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:36.506594    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:37.592119 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:37.592084    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d69556c9f-7kz25"]
Sep 10 20:59:39.087029 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:39.086915    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d69556c9f-7kz25"]
Sep 10 20:59:39.171847 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:39.171764    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj" event={"ID":"ce584ac9-fc93-43da-8cb2-2f7082a2b59a","Type":"ContainerStarted","Data":"49701c0e4448e3ba39df6a0121f94f5447cff622582a236277789cee56945176"}
Sep 10 20:59:39.171847 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:39.171813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj" event={"ID":"ce584ac9-fc93-43da-8cb2-2f7082a2b59a","Type":"ContainerStarted","Data":"0c18c6db70cddda96522c4b8775cff67acfa055c0edbd163344d1f783dc57e3c"}
Sep 10 20:59:39.173728 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:39.173703    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-684c958f79-l44l5" event={"ID":"c4adbb2b-130d-4a30-beb9-d8e8c438acc9","Type":"ContainerStarted","Data":"b54bc41211ed797ac94ecb738b5030c7fd0cc76995ce2c5aa7fb80a9f516de3f"}
Sep 10 20:59:39.173837 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:39.173773    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-684c958f79-l44l5" event={"ID":"c4adbb2b-130d-4a30-beb9-d8e8c438acc9","Type":"ContainerStarted","Data":"3e7aead0d3988981637567234a060468bcbdb5ebb2d79d3213ddc204f87a5ebe"}
Sep 10 20:59:39.177433 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:39.176651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w" event={"ID":"188ff6df-fa09-4db5-a54c-82fb0bb440bc","Type":"ContainerStarted","Data":"1b4b2abd350bbcdef9cf641810d2683588827cbe0fa13be3c6e06871970c6360"}
Sep 10 20:59:39.178085 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:39.178058    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d69556c9f-7kz25" event={"ID":"3129da09-6cf1-49cf-a775-166ef15e9f3d","Type":"ContainerStarted","Data":"b305db82474fb2495571c20327bed85d21482d4c43690eea3eeda63c98200870"}
Sep 10 20:59:39.180560 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:39.180524    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerID="fbf50c561d0fccf7dc23589f34dcc8c8c615f168077a5757b2be442c945564f4" exitCode=0
Sep 10 20:59:39.180651 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:39.180571    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e733ef7-73f2-40cd-bfeb-020645b2b3ee","Type":"ContainerDied","Data":"fbf50c561d0fccf7dc23589f34dcc8c8c615f168077a5757b2be442c945564f4"}
Sep 10 20:59:39.205041 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:39.204891    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w" podStartSLOduration=4.689423843 podStartE2EDuration="9.204868476s" totalImagesPullingTime="4.515444633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:39.199799191 +0000 UTC m=+120.175459240" watchObservedRunningTime="2026-09-10 20:59:39.204868476 +0000 UTC m=+120.180528524"
Sep 10 20:59:39.307415 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:39.305145    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f469fbc67-26fpg"]
Sep 10 20:59:39.311845 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:39.311778    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 20:59:40.189342 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:40.189262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj" event={"ID":"ce584ac9-fc93-43da-8cb2-2f7082a2b59a","Type":"ContainerStarted","Data":"d943b6db3662e048e346cb62e0cec5698f9f7f95f3df3c0ccbd44e6367dd9705"}
Sep 10 20:59:40.192798 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:40.192666    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-684c958f79-l44l5" event={"ID":"c4adbb2b-130d-4a30-beb9-d8e8c438acc9","Type":"ContainerStarted","Data":"5b0a7b1f183c0e48d4c9016d2bf1bb714cf256e96da8a1107cb52a3ca5780727"}
Sep 10 20:59:40.196447 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:40.196410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d69556c9f-7kz25" event={"ID":"3129da09-6cf1-49cf-a775-166ef15e9f3d","Type":"ContainerStarted","Data":"d8432aa3eb311c638532267e8848d359ff1aa0a24af95b05e592053ce818450b"}
Sep 10 20:59:40.286868 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:40.286810    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqvpj" podStartSLOduration=9.775867188 podStartE2EDuration="14.286789408s" totalImagesPullingTime="4.51092222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:40.238423046 +0000 UTC m=+121.214083094" watchObservedRunningTime="2026-09-10 20:59:40.286789408 +0000 UTC m=+121.262449454"
Sep 10 20:59:40.287458 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:40.287414    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d69556c9f-7kz25" podStartSLOduration=4.287399914 podStartE2EDuration="4.287399914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:40.286249201 +0000 UTC m=+121.261909249" watchObservedRunningTime="2026-09-10 20:59:40.287399914 +0000 UTC m=+121.263059961"
Sep 10 20:59:42.218529 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:42.218438    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-684c958f79-l44l5" event={"ID":"c4adbb2b-130d-4a30-beb9-d8e8c438acc9","Type":"ContainerStarted","Data":"a177362e1a27e0c9a15783ffb36a542e5a906ff2a27d462fcb454d47657e95a8"}
Sep 10 20:59:42.218950 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:42.218543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-684c958f79-l44l5" event={"ID":"c4adbb2b-130d-4a30-beb9-d8e8c438acc9","Type":"ContainerStarted","Data":"71defbddd518693db9eda988ac466de374a3927e257740bebde1be759de56710"}
Sep 10 20:59:42.218950 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:42.218585    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-684c958f79-l44l5" event={"ID":"c4adbb2b-130d-4a30-beb9-d8e8c438acc9","Type":"ContainerStarted","Data":"f3b27668b65580395e08705599fc658e6d0839404069d1e66d5981f3aa4f5025"}
Sep 10 20:59:42.218950 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:42.218836    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:42.223645 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:42.223610    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e733ef7-73f2-40cd-bfeb-020645b2b3ee","Type":"ContainerStarted","Data":"ee3bf62ce7c13251d84322174230b99aa616b34f6b98088142fd501de43b0f0a"}
Sep 10 20:59:42.223784 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:42.223675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e733ef7-73f2-40cd-bfeb-020645b2b3ee","Type":"ContainerStarted","Data":"492f5934570d670164a4e5d23f77d44c08fb19aee995eecd104c7aa334da7c84"}
Sep 10 20:59:42.223784 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:42.223707    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e733ef7-73f2-40cd-bfeb-020645b2b3ee","Type":"ContainerStarted","Data":"e94e23c96b3132649ffad134c675ef219e3f6fe6e26b29b83ac9095afe1c9d71"}
Sep 10 20:59:42.223784 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:42.223740    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e733ef7-73f2-40cd-bfeb-020645b2b3ee","Type":"ContainerStarted","Data":"e765e388a53c47581bd2edc173da9d189db46a6c4963c9445acf1e3c9428029c"}
Sep 10 20:59:42.223784 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:42.223772    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e733ef7-73f2-40cd-bfeb-020645b2b3ee","Type":"ContainerStarted","Data":"085a47919657564f8a8a3dbeea059cfbda3b2f10bb79270141325fc1e8ec9f08"}
Sep 10 20:59:42.223973 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:42.223807    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e733ef7-73f2-40cd-bfeb-020645b2b3ee","Type":"ContainerStarted","Data":"fa53573ae0be1016ce0ad3aaf696d24714a37599d2043964bc2084abd9b99db9"}
Sep 10 20:59:42.244407 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:42.244362    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-684c958f79-l44l5" podStartSLOduration=7.679994987 podStartE2EDuration="14.244345926s" totalImagesPullingTime="6.564350939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:42.243495683 +0000 UTC m=+123.219155730" watchObservedRunningTime="2026-09-10 20:59:42.244345926 +0000 UTC m=+123.220005973"
Sep 10 20:59:42.279337 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:42.279290    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=8.572279848 podStartE2EDuration="15.27927536s" totalImagesPullingTime="6.689020709s" totalInitContainerRuntime="17.974803ms" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:42.277433371 +0000 UTC m=+123.253093418" watchObservedRunningTime="2026-09-10 20:59:42.27927536 +0000 UTC m=+123.254935408"
Sep 10 20:59:46.507494 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:46.507462    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 20:59:48.234431 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:48.234403    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-684c958f79-l44l5"
Sep 10 20:59:48.268933 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:48.268903    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-565f5f457b-phlc5"]
Sep 10 20:59:49.284335 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:49.284305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs\") pod \"network-metrics-daemon-4r2gd\" (UID: \"278ac5cb-23c6-4748-b751-d5f3c53fdb23\") " pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:59:49.286547 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:49.286529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278ac5cb-23c6-4748-b751-d5f3c53fdb23-metrics-certs\") pod \"network-metrics-daemon-4r2gd\" (UID: \"278ac5cb-23c6-4748-b751-d5f3c53fdb23\") " pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:59:49.387689 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:49.387666    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tjcr5\""
Sep 10 20:59:49.395656 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:49.395639    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r2gd"
Sep 10 20:59:49.699629 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:49.699552    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4r2gd"]
Sep 10 20:59:49.703148 ip-10-0-134-6 kubenswrapper[2581]: W0910 20:59:49.703111    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278ac5cb_23c6_4748_b751_d5f3c53fdb23.slice/crio-d03c8901c99223ee2f16e88a40a4c3910c117981bdcca7a4571744d1921d47f9 WatchSource:0}: Error finding container d03c8901c99223ee2f16e88a40a4c3910c117981bdcca7a4571744d1921d47f9: Status 404 returned error can't find the container with id d03c8901c99223ee2f16e88a40a4c3910c117981bdcca7a4571744d1921d47f9
Sep 10 20:59:50.256611 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:50.256574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4r2gd" event={"ID":"278ac5cb-23c6-4748-b751-d5f3c53fdb23","Type":"ContainerStarted","Data":"d03c8901c99223ee2f16e88a40a4c3910c117981bdcca7a4571744d1921d47f9"}
Sep 10 20:59:50.849725 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:50.849693    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:50.850069 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:50.849754    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 20:59:51.262166 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:51.262135    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4r2gd" event={"ID":"278ac5cb-23c6-4748-b751-d5f3c53fdb23","Type":"ContainerStarted","Data":"d8d1a5b37fb91a05292ca050d03299d8023fb6f36ea4bee870d41adf8aadf9a3"}
Sep 10 20:59:52.267819 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:52.267786    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4r2gd" event={"ID":"278ac5cb-23c6-4748-b751-d5f3c53fdb23","Type":"ContainerStarted","Data":"2e09f910db213f65aa004bbde547be700fb497b6568cbdc65968adfa8c7dc794"}
Sep 10 20:59:52.286760 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:52.286706    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4r2gd" podStartSLOduration=130.913226831 podStartE2EDuration="2m12.286689516s" totalImagesPullingTime="1.373462685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:52.285661378 +0000 UTC m=+133.261321425" watchObservedRunningTime="2026-09-10 20:59:52.286689516 +0000 UTC m=+133.262349563"
Sep 10 20:59:56.202522 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.202479    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5946d7ffd9-5mhnj" podUID="7be70674-f7fc-4d4f-a712-c61c7e651c3a" containerName="console" containerID="cri-o://dd777f83413b78d4439cdb8deecf4aad4b073be2d9227e218a28289ffcb2c7dd" gracePeriod=15
Sep 10 20:59:56.476168 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.476112    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:56.544381 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.544355    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-config\") pod \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") "
Sep 10 20:59:56.544518 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.544420    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-service-ca\") pod \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") "
Sep 10 20:59:56.544518 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.544441    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-oauth-config\") pod \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") "
Sep 10 20:59:56.544518 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.544468    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-oauth-serving-cert\") pod \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") "
Sep 10 20:59:56.544518 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.544512    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-serving-cert\") pod \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") "
Sep 10 20:59:56.544739 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.544536    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hztm2\" (UniqueName: \"kubernetes.io/projected/7be70674-f7fc-4d4f-a712-c61c7e651c3a-kube-api-access-hztm2\") pod \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\" (UID: \"7be70674-f7fc-4d4f-a712-c61c7e651c3a\") "
Sep 10 20:59:56.544893 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.544860    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-config" (OuterVolumeSpecName: "console-config") pod "7be70674-f7fc-4d4f-a712-c61c7e651c3a" (UID: "7be70674-f7fc-4d4f-a712-c61c7e651c3a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:59:56.545004 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.544981    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-service-ca" (OuterVolumeSpecName: "service-ca") pod "7be70674-f7fc-4d4f-a712-c61c7e651c3a" (UID: "7be70674-f7fc-4d4f-a712-c61c7e651c3a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:59:56.545051 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.544998    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7be70674-f7fc-4d4f-a712-c61c7e651c3a" (UID: "7be70674-f7fc-4d4f-a712-c61c7e651c3a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:59:56.546682 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.546648    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7be70674-f7fc-4d4f-a712-c61c7e651c3a" (UID: "7be70674-f7fc-4d4f-a712-c61c7e651c3a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:59:56.546786 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.546689    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7be70674-f7fc-4d4f-a712-c61c7e651c3a" (UID: "7be70674-f7fc-4d4f-a712-c61c7e651c3a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:59:56.546786 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.546726    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be70674-f7fc-4d4f-a712-c61c7e651c3a-kube-api-access-hztm2" (OuterVolumeSpecName: "kube-api-access-hztm2") pod "7be70674-f7fc-4d4f-a712-c61c7e651c3a" (UID: "7be70674-f7fc-4d4f-a712-c61c7e651c3a"). InnerVolumeSpecName "kube-api-access-hztm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:59:56.645218 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.645168    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-service-ca\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 20:59:56.645218 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.645194    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-oauth-config\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 20:59:56.645218 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.645223    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-oauth-serving-cert\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 20:59:56.645402 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.645231    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-serving-cert\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 20:59:56.645402 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.645240    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hztm2\" (UniqueName: \"kubernetes.io/projected/7be70674-f7fc-4d4f-a712-c61c7e651c3a-kube-api-access-hztm2\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 20:59:56.645402 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:56.645248    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7be70674-f7fc-4d4f-a712-c61c7e651c3a-console-config\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 20:59:57.288268 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:57.288236    2581 generic.go:356] "Generic (PLEG): container finished" podID="7be70674-f7fc-4d4f-a712-c61c7e651c3a" containerID="dd777f83413b78d4439cdb8deecf4aad4b073be2d9227e218a28289ffcb2c7dd" exitCode=0
Sep 10 20:59:57.288667 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:57.288284    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5946d7ffd9-5mhnj"
Sep 10 20:59:57.288667 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:57.288303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5946d7ffd9-5mhnj" event={"ID":"7be70674-f7fc-4d4f-a712-c61c7e651c3a","Type":"ContainerDied","Data":"dd777f83413b78d4439cdb8deecf4aad4b073be2d9227e218a28289ffcb2c7dd"}
Sep 10 20:59:57.288667 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:57.288343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5946d7ffd9-5mhnj" event={"ID":"7be70674-f7fc-4d4f-a712-c61c7e651c3a","Type":"ContainerDied","Data":"e03c775cf5b5ef9c687df10fe0b5700ccb6f50f635f621ac23355962f1647855"}
Sep 10 20:59:57.288667 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:57.288362    2581 scope.go:117] "RemoveContainer" containerID="dd777f83413b78d4439cdb8deecf4aad4b073be2d9227e218a28289ffcb2c7dd"
Sep 10 20:59:57.301471 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:57.301456    2581 scope.go:117] "RemoveContainer" containerID="dd777f83413b78d4439cdb8deecf4aad4b073be2d9227e218a28289ffcb2c7dd"
Sep 10 20:59:57.301732 ip-10-0-134-6 kubenswrapper[2581]: E0910 20:59:57.301709    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd777f83413b78d4439cdb8deecf4aad4b073be2d9227e218a28289ffcb2c7dd\": container with ID starting with dd777f83413b78d4439cdb8deecf4aad4b073be2d9227e218a28289ffcb2c7dd not found: ID does not exist" containerID="dd777f83413b78d4439cdb8deecf4aad4b073be2d9227e218a28289ffcb2c7dd"
Sep 10 20:59:57.301803 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:57.301745    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd777f83413b78d4439cdb8deecf4aad4b073be2d9227e218a28289ffcb2c7dd"} err="failed to get container status \"dd777f83413b78d4439cdb8deecf4aad4b073be2d9227e218a28289ffcb2c7dd\": rpc error: code = NotFound desc = could not find container \"dd777f83413b78d4439cdb8deecf4aad4b073be2d9227e218a28289ffcb2c7dd\": container with ID starting with dd777f83413b78d4439cdb8deecf4aad4b073be2d9227e218a28289ffcb2c7dd not found: ID does not exist"
Sep 10 20:59:57.309753 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:57.309727    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5946d7ffd9-5mhnj"]
Sep 10 20:59:57.314461 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:57.314276    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5946d7ffd9-5mhnj"]
Sep 10 20:59:57.573724 ip-10-0-134-6 kubenswrapper[2581]: I0910 20:59:57.573633    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7be70674-f7fc-4d4f-a712-c61c7e651c3a" path="/var/lib/kubelet/pods/7be70674-f7fc-4d4f-a712-c61c7e651c3a/volumes"
Sep 10 21:00:04.336082 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.336021    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-f469fbc67-26fpg" podUID="dd68b468-3b12-4738-b092-1f3e6e3c3356" containerName="registry" containerID="cri-o://8a9c99d1cb5babcc4a4fd6cf25dc7bac2ba2dfd94c835be775e5686a1f1216c4" gracePeriod=30
Sep 10 21:00:04.565241 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.565219    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 21:00:04.714183 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.714104    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd68b468-3b12-4738-b092-1f3e6e3c3356-trusted-ca\") pod \"dd68b468-3b12-4738-b092-1f3e6e3c3356\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") "
Sep 10 21:00:04.714183 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.714150    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd68b468-3b12-4738-b092-1f3e6e3c3356-image-registry-private-configuration\") pod \"dd68b468-3b12-4738-b092-1f3e6e3c3356\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") "
Sep 10 21:00:04.714183 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.714172    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd68b468-3b12-4738-b092-1f3e6e3c3356-installation-pull-secrets\") pod \"dd68b468-3b12-4738-b092-1f3e6e3c3356\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") "
Sep 10 21:00:04.714426 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.714251    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls\") pod \"dd68b468-3b12-4738-b092-1f3e6e3c3356\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") "
Sep 10 21:00:04.714426 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.714296    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-certificates\") pod \"dd68b468-3b12-4738-b092-1f3e6e3c3356\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") "
Sep 10 21:00:04.714426 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.714342    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd68b468-3b12-4738-b092-1f3e6e3c3356-ca-trust-extracted\") pod \"dd68b468-3b12-4738-b092-1f3e6e3c3356\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") "
Sep 10 21:00:04.714426 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.714368    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgkgw\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-kube-api-access-lgkgw\") pod \"dd68b468-3b12-4738-b092-1f3e6e3c3356\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") "
Sep 10 21:00:04.714426 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.714405    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-bound-sa-token\") pod \"dd68b468-3b12-4738-b092-1f3e6e3c3356\" (UID: \"dd68b468-3b12-4738-b092-1f3e6e3c3356\") "
Sep 10 21:00:04.714725 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.714685    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd68b468-3b12-4738-b092-1f3e6e3c3356-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "dd68b468-3b12-4738-b092-1f3e6e3c3356" (UID: "dd68b468-3b12-4738-b092-1f3e6e3c3356"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:04.714829 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.714803    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "dd68b468-3b12-4738-b092-1f3e6e3c3356" (UID: "dd68b468-3b12-4738-b092-1f3e6e3c3356"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:04.717105 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.717076    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd68b468-3b12-4738-b092-1f3e6e3c3356-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "dd68b468-3b12-4738-b092-1f3e6e3c3356" (UID: "dd68b468-3b12-4738-b092-1f3e6e3c3356"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:04.717241 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.717155    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "dd68b468-3b12-4738-b092-1f3e6e3c3356" (UID: "dd68b468-3b12-4738-b092-1f3e6e3c3356"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:00:04.717241 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.717164    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "dd68b468-3b12-4738-b092-1f3e6e3c3356" (UID: "dd68b468-3b12-4738-b092-1f3e6e3c3356"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:00:04.717241 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.717180    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd68b468-3b12-4738-b092-1f3e6e3c3356-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "dd68b468-3b12-4738-b092-1f3e6e3c3356" (UID: "dd68b468-3b12-4738-b092-1f3e6e3c3356"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:04.717406 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.717270    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-kube-api-access-lgkgw" (OuterVolumeSpecName: "kube-api-access-lgkgw") pod "dd68b468-3b12-4738-b092-1f3e6e3c3356" (UID: "dd68b468-3b12-4738-b092-1f3e6e3c3356"). InnerVolumeSpecName "kube-api-access-lgkgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:00:04.722176 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.722147    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd68b468-3b12-4738-b092-1f3e6e3c3356-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "dd68b468-3b12-4738-b092-1f3e6e3c3356" (UID: "dd68b468-3b12-4738-b092-1f3e6e3c3356"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:00:04.815522 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.815498    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-bound-sa-token\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:04.815522 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.815520    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd68b468-3b12-4738-b092-1f3e6e3c3356-trusted-ca\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:04.815653 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.815529    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd68b468-3b12-4738-b092-1f3e6e3c3356-image-registry-private-configuration\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:04.815653 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.815539    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd68b468-3b12-4738-b092-1f3e6e3c3356-installation-pull-secrets\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:04.815653 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.815548    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-tls\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:04.815653 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.815556    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd68b468-3b12-4738-b092-1f3e6e3c3356-registry-certificates\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:04.815653 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.815564    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd68b468-3b12-4738-b092-1f3e6e3c3356-ca-trust-extracted\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:04.815653 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:04.815572    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgkgw\" (UniqueName: \"kubernetes.io/projected/dd68b468-3b12-4738-b092-1f3e6e3c3356-kube-api-access-lgkgw\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:05.221577 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.221537    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d69556c9f-7kz25" podUID="3129da09-6cf1-49cf-a775-166ef15e9f3d" containerName="console" containerID="cri-o://d8432aa3eb311c638532267e8848d359ff1aa0a24af95b05e592053ce818450b" gracePeriod=15
Sep 10 21:00:05.320531 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.320503    2581 generic.go:356] "Generic (PLEG): container finished" podID="dd68b468-3b12-4738-b092-1f3e6e3c3356" containerID="8a9c99d1cb5babcc4a4fd6cf25dc7bac2ba2dfd94c835be775e5686a1f1216c4" exitCode=0
Sep 10 21:00:05.320677 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.320539    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f469fbc67-26fpg" event={"ID":"dd68b468-3b12-4738-b092-1f3e6e3c3356","Type":"ContainerDied","Data":"8a9c99d1cb5babcc4a4fd6cf25dc7bac2ba2dfd94c835be775e5686a1f1216c4"}
Sep 10 21:00:05.320677 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.320562    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f469fbc67-26fpg"
Sep 10 21:00:05.320677 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.320574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f469fbc67-26fpg" event={"ID":"dd68b468-3b12-4738-b092-1f3e6e3c3356","Type":"ContainerDied","Data":"703262f5a1cdb719618852d45bb23debe92261231f42b90dca321f564ae5e80e"}
Sep 10 21:00:05.320677 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.320592    2581 scope.go:117] "RemoveContainer" containerID="8a9c99d1cb5babcc4a4fd6cf25dc7bac2ba2dfd94c835be775e5686a1f1216c4"
Sep 10 21:00:05.335244 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.335224    2581 scope.go:117] "RemoveContainer" containerID="8a9c99d1cb5babcc4a4fd6cf25dc7bac2ba2dfd94c835be775e5686a1f1216c4"
Sep 10 21:00:05.335506 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:00:05.335486    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9c99d1cb5babcc4a4fd6cf25dc7bac2ba2dfd94c835be775e5686a1f1216c4\": container with ID starting with 8a9c99d1cb5babcc4a4fd6cf25dc7bac2ba2dfd94c835be775e5686a1f1216c4 not found: ID does not exist" containerID="8a9c99d1cb5babcc4a4fd6cf25dc7bac2ba2dfd94c835be775e5686a1f1216c4"
Sep 10 21:00:05.335578 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.335517    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9c99d1cb5babcc4a4fd6cf25dc7bac2ba2dfd94c835be775e5686a1f1216c4"} err="failed to get container status \"8a9c99d1cb5babcc4a4fd6cf25dc7bac2ba2dfd94c835be775e5686a1f1216c4\": rpc error: code = NotFound desc = could not find container \"8a9c99d1cb5babcc4a4fd6cf25dc7bac2ba2dfd94c835be775e5686a1f1216c4\": container with ID starting with 8a9c99d1cb5babcc4a4fd6cf25dc7bac2ba2dfd94c835be775e5686a1f1216c4 not found: ID does not exist"
Sep 10 21:00:05.344706 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.344686    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f469fbc67-26fpg"]
Sep 10 21:00:05.349576 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.349558    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f469fbc67-26fpg"]
Sep 10 21:00:05.467955 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.467934    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 21:00:05.573136 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.573107    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd68b468-3b12-4738-b092-1f3e6e3c3356" path="/var/lib/kubelet/pods/dd68b468-3b12-4738-b092-1f3e6e3c3356/volumes"
Sep 10 21:00:05.622254 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.622228    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-oauth-serving-cert\") pod \"3129da09-6cf1-49cf-a775-166ef15e9f3d\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") "
Sep 10 21:00:05.622254 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.622255    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-service-ca\") pod \"3129da09-6cf1-49cf-a775-166ef15e9f3d\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") "
Sep 10 21:00:05.622421 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.622284    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcqcq\" (UniqueName: \"kubernetes.io/projected/3129da09-6cf1-49cf-a775-166ef15e9f3d-kube-api-access-hcqcq\") pod \"3129da09-6cf1-49cf-a775-166ef15e9f3d\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") "
Sep 10 21:00:05.622487 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.622469    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-oauth-config\") pod \"3129da09-6cf1-49cf-a775-166ef15e9f3d\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") "
Sep 10 21:00:05.622531 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.622517    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-config\") pod \"3129da09-6cf1-49cf-a775-166ef15e9f3d\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") "
Sep 10 21:00:05.622586 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.622562    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-serving-cert\") pod \"3129da09-6cf1-49cf-a775-166ef15e9f3d\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") "
Sep 10 21:00:05.622640 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.622588    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-trusted-ca-bundle\") pod \"3129da09-6cf1-49cf-a775-166ef15e9f3d\" (UID: \"3129da09-6cf1-49cf-a775-166ef15e9f3d\") "
Sep 10 21:00:05.622640 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.622620    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3129da09-6cf1-49cf-a775-166ef15e9f3d" (UID: "3129da09-6cf1-49cf-a775-166ef15e9f3d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:05.622741 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.622648    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-service-ca" (OuterVolumeSpecName: "service-ca") pod "3129da09-6cf1-49cf-a775-166ef15e9f3d" (UID: "3129da09-6cf1-49cf-a775-166ef15e9f3d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:05.622846 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.622821    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-oauth-serving-cert\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:05.622964 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.622847    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-service-ca\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:05.622964 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.622850    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-config" (OuterVolumeSpecName: "console-config") pod "3129da09-6cf1-49cf-a775-166ef15e9f3d" (UID: "3129da09-6cf1-49cf-a775-166ef15e9f3d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:05.623072 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.623021    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3129da09-6cf1-49cf-a775-166ef15e9f3d" (UID: "3129da09-6cf1-49cf-a775-166ef15e9f3d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:05.624590 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.624568    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3129da09-6cf1-49cf-a775-166ef15e9f3d-kube-api-access-hcqcq" (OuterVolumeSpecName: "kube-api-access-hcqcq") pod "3129da09-6cf1-49cf-a775-166ef15e9f3d" (UID: "3129da09-6cf1-49cf-a775-166ef15e9f3d"). InnerVolumeSpecName "kube-api-access-hcqcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:00:05.624716 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.624697    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3129da09-6cf1-49cf-a775-166ef15e9f3d" (UID: "3129da09-6cf1-49cf-a775-166ef15e9f3d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:05.624769 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.624717    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3129da09-6cf1-49cf-a775-166ef15e9f3d" (UID: "3129da09-6cf1-49cf-a775-166ef15e9f3d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:05.723990 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.723960    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-config\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:05.723990 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.723985    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-serving-cert\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:05.723990 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.723994    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3129da09-6cf1-49cf-a775-166ef15e9f3d-trusted-ca-bundle\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:05.724172 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.724003    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcqcq\" (UniqueName: \"kubernetes.io/projected/3129da09-6cf1-49cf-a775-166ef15e9f3d-kube-api-access-hcqcq\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:05.724172 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:05.724012    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3129da09-6cf1-49cf-a775-166ef15e9f3d-console-oauth-config\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:06.327022 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:06.326983    2581 generic.go:356] "Generic (PLEG): container finished" podID="3129da09-6cf1-49cf-a775-166ef15e9f3d" containerID="d8432aa3eb311c638532267e8848d359ff1aa0a24af95b05e592053ce818450b" exitCode=0
Sep 10 21:00:06.327167 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:06.327033    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d69556c9f-7kz25"
Sep 10 21:00:06.327167 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:06.327073    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d69556c9f-7kz25" event={"ID":"3129da09-6cf1-49cf-a775-166ef15e9f3d","Type":"ContainerDied","Data":"d8432aa3eb311c638532267e8848d359ff1aa0a24af95b05e592053ce818450b"}
Sep 10 21:00:06.327167 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:06.327127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d69556c9f-7kz25" event={"ID":"3129da09-6cf1-49cf-a775-166ef15e9f3d","Type":"ContainerDied","Data":"b305db82474fb2495571c20327bed85d21482d4c43690eea3eeda63c98200870"}
Sep 10 21:00:06.327167 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:06.327148    2581 scope.go:117] "RemoveContainer" containerID="d8432aa3eb311c638532267e8848d359ff1aa0a24af95b05e592053ce818450b"
Sep 10 21:00:06.336173 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:06.336156    2581 scope.go:117] "RemoveContainer" containerID="d8432aa3eb311c638532267e8848d359ff1aa0a24af95b05e592053ce818450b"
Sep 10 21:00:06.336447 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:00:06.336429    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8432aa3eb311c638532267e8848d359ff1aa0a24af95b05e592053ce818450b\": container with ID starting with d8432aa3eb311c638532267e8848d359ff1aa0a24af95b05e592053ce818450b not found: ID does not exist" containerID="d8432aa3eb311c638532267e8848d359ff1aa0a24af95b05e592053ce818450b"
Sep 10 21:00:06.336498 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:06.336455    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8432aa3eb311c638532267e8848d359ff1aa0a24af95b05e592053ce818450b"} err="failed to get container status \"d8432aa3eb311c638532267e8848d359ff1aa0a24af95b05e592053ce818450b\": rpc error: code = NotFound desc = could not find container \"d8432aa3eb311c638532267e8848d359ff1aa0a24af95b05e592053ce818450b\": container with ID starting with d8432aa3eb311c638532267e8848d359ff1aa0a24af95b05e592053ce818450b not found: ID does not exist"
Sep 10 21:00:06.348557 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:06.348532    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d69556c9f-7kz25"]
Sep 10 21:00:06.352681 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:06.352654    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d69556c9f-7kz25"]
Sep 10 21:00:07.573053 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:07.573018    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3129da09-6cf1-49cf-a775-166ef15e9f3d" path="/var/lib/kubelet/pods/3129da09-6cf1-49cf-a775-166ef15e9f3d/volumes"
Sep 10 21:00:10.855184 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:10.855155    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 21:00:10.858965 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:10.858942    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7bcbdfdfbf-85x6w"
Sep 10 21:00:13.288256 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.288184    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-565f5f457b-phlc5" podUID="89dded4f-6ba3-47b6-9134-307795dee576" containerName="console" containerID="cri-o://1199bb4496070c7602ad744528b6723b6a897658f4f4efbfcb4c27220d9e0d31" gracePeriod=15
Sep 10 21:00:13.575213 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.575171    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 21:00:13.691014 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.690985    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-service-ca\") pod \"89dded4f-6ba3-47b6-9134-307795dee576\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") "
Sep 10 21:00:13.691219 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.691037    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p52g\" (UniqueName: \"kubernetes.io/projected/89dded4f-6ba3-47b6-9134-307795dee576-kube-api-access-8p52g\") pod \"89dded4f-6ba3-47b6-9134-307795dee576\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") "
Sep 10 21:00:13.691219 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.691074    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-trusted-ca-bundle\") pod \"89dded4f-6ba3-47b6-9134-307795dee576\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") "
Sep 10 21:00:13.691219 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.691103    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-oauth-serving-cert\") pod \"89dded4f-6ba3-47b6-9134-307795dee576\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") "
Sep 10 21:00:13.691219 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.691140    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-console-config\") pod \"89dded4f-6ba3-47b6-9134-307795dee576\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") "
Sep 10 21:00:13.691219 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.691186    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89dded4f-6ba3-47b6-9134-307795dee576-console-oauth-config\") pod \"89dded4f-6ba3-47b6-9134-307795dee576\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") "
Sep 10 21:00:13.691464 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.691244    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89dded4f-6ba3-47b6-9134-307795dee576-console-serving-cert\") pod \"89dded4f-6ba3-47b6-9134-307795dee576\" (UID: \"89dded4f-6ba3-47b6-9134-307795dee576\") "
Sep 10 21:00:13.691516 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.691494    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-service-ca" (OuterVolumeSpecName: "service-ca") pod "89dded4f-6ba3-47b6-9134-307795dee576" (UID: "89dded4f-6ba3-47b6-9134-307795dee576"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:13.691567 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.691526    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "89dded4f-6ba3-47b6-9134-307795dee576" (UID: "89dded4f-6ba3-47b6-9134-307795dee576"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:13.691567 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.691553    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "89dded4f-6ba3-47b6-9134-307795dee576" (UID: "89dded4f-6ba3-47b6-9134-307795dee576"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:13.691670 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.691618    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-console-config" (OuterVolumeSpecName: "console-config") pod "89dded4f-6ba3-47b6-9134-307795dee576" (UID: "89dded4f-6ba3-47b6-9134-307795dee576"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:13.693406 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.693374    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89dded4f-6ba3-47b6-9134-307795dee576-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "89dded4f-6ba3-47b6-9134-307795dee576" (UID: "89dded4f-6ba3-47b6-9134-307795dee576"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:13.693406 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.693393    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89dded4f-6ba3-47b6-9134-307795dee576-kube-api-access-8p52g" (OuterVolumeSpecName: "kube-api-access-8p52g") pod "89dded4f-6ba3-47b6-9134-307795dee576" (UID: "89dded4f-6ba3-47b6-9134-307795dee576"). InnerVolumeSpecName "kube-api-access-8p52g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:00:13.693544 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.693394    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89dded4f-6ba3-47b6-9134-307795dee576-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "89dded4f-6ba3-47b6-9134-307795dee576" (UID: "89dded4f-6ba3-47b6-9134-307795dee576"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:13.792543 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.792476    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-trusted-ca-bundle\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:13.792543 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.792501    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-oauth-serving-cert\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:13.792543 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.792511    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-console-config\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:13.792543 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.792520    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89dded4f-6ba3-47b6-9134-307795dee576-console-oauth-config\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:13.792543 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.792528    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89dded4f-6ba3-47b6-9134-307795dee576-console-serving-cert\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:13.792543 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.792537    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89dded4f-6ba3-47b6-9134-307795dee576-service-ca\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:13.792543 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:13.792545    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p52g\" (UniqueName: \"kubernetes.io/projected/89dded4f-6ba3-47b6-9134-307795dee576-kube-api-access-8p52g\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:14.360051 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:14.360017    2581 generic.go:356] "Generic (PLEG): container finished" podID="89dded4f-6ba3-47b6-9134-307795dee576" containerID="1199bb4496070c7602ad744528b6723b6a897658f4f4efbfcb4c27220d9e0d31" exitCode=0
Sep 10 21:00:14.360527 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:14.360066    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565f5f457b-phlc5"
Sep 10 21:00:14.360527 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:14.360063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565f5f457b-phlc5" event={"ID":"89dded4f-6ba3-47b6-9134-307795dee576","Type":"ContainerDied","Data":"1199bb4496070c7602ad744528b6723b6a897658f4f4efbfcb4c27220d9e0d31"}
Sep 10 21:00:14.360527 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:14.360234    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565f5f457b-phlc5" event={"ID":"89dded4f-6ba3-47b6-9134-307795dee576","Type":"ContainerDied","Data":"be69c81ade14f7ab1092a6bab69a3b3348f97a0d88d27285855ddd0dfcca5d24"}
Sep 10 21:00:14.360527 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:14.360273    2581 scope.go:117] "RemoveContainer" containerID="1199bb4496070c7602ad744528b6723b6a897658f4f4efbfcb4c27220d9e0d31"
Sep 10 21:00:14.368633 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:14.368615    2581 scope.go:117] "RemoveContainer" containerID="1199bb4496070c7602ad744528b6723b6a897658f4f4efbfcb4c27220d9e0d31"
Sep 10 21:00:14.368881 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:00:14.368862    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1199bb4496070c7602ad744528b6723b6a897658f4f4efbfcb4c27220d9e0d31\": container with ID starting with 1199bb4496070c7602ad744528b6723b6a897658f4f4efbfcb4c27220d9e0d31 not found: ID does not exist" containerID="1199bb4496070c7602ad744528b6723b6a897658f4f4efbfcb4c27220d9e0d31"
Sep 10 21:00:14.368918 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:14.368888    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1199bb4496070c7602ad744528b6723b6a897658f4f4efbfcb4c27220d9e0d31"} err="failed to get container status \"1199bb4496070c7602ad744528b6723b6a897658f4f4efbfcb4c27220d9e0d31\": rpc error: code = NotFound desc = could not find container \"1199bb4496070c7602ad744528b6723b6a897658f4f4efbfcb4c27220d9e0d31\": container with ID starting with 1199bb4496070c7602ad744528b6723b6a897658f4f4efbfcb4c27220d9e0d31 not found: ID does not exist"
Sep 10 21:00:14.380805 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:14.380778    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-565f5f457b-phlc5"]
Sep 10 21:00:14.384384 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:14.384364    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-565f5f457b-phlc5"]
Sep 10 21:00:15.572407 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:15.572372    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89dded4f-6ba3-47b6-9134-307795dee576" path="/var/lib/kubelet/pods/89dded4f-6ba3-47b6-9134-307795dee576/volumes"
Sep 10 21:00:46.703025 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.702995    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 21:00:46.729364 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.729336    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58b4874f44-5sm6k"]
Sep 10 21:00:46.729640 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.729627    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7be70674-f7fc-4d4f-a712-c61c7e651c3a" containerName="console"
Sep 10 21:00:46.729685 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.729642    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7be70674-f7fc-4d4f-a712-c61c7e651c3a" containerName="console"
Sep 10 21:00:46.729685 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.729652    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89dded4f-6ba3-47b6-9134-307795dee576" containerName="console"
Sep 10 21:00:46.729685 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.729657    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89dded4f-6ba3-47b6-9134-307795dee576" containerName="console"
Sep 10 21:00:46.729685 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.729665    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd68b468-3b12-4738-b092-1f3e6e3c3356" containerName="registry"
Sep 10 21:00:46.729685 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.729671    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd68b468-3b12-4738-b092-1f3e6e3c3356" containerName="registry"
Sep 10 21:00:46.729685 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.729678    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3129da09-6cf1-49cf-a775-166ef15e9f3d" containerName="console"
Sep 10 21:00:46.729685 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.729683    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3129da09-6cf1-49cf-a775-166ef15e9f3d" containerName="console"
Sep 10 21:00:46.729883 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.729735    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3129da09-6cf1-49cf-a775-166ef15e9f3d" containerName="console"
Sep 10 21:00:46.729883 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.729742    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="89dded4f-6ba3-47b6-9134-307795dee576" containerName="console"
Sep 10 21:00:46.729883 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.729750    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7be70674-f7fc-4d4f-a712-c61c7e651c3a" containerName="console"
Sep 10 21:00:46.729883 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.729757    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd68b468-3b12-4738-b092-1f3e6e3c3356" containerName="registry"
Sep 10 21:00:46.733970 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.733951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.736807 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.736779    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 21:00:46.736919 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.736905    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 21:00:46.737774 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.737760    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 21:00:46.737867 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.737847    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 21:00:46.737978 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.737965    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 21:00:46.738219 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.738178    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-66nzk\""
Sep 10 21:00:46.743628 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.743612    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 21:00:46.752399 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.752378    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58b4874f44-5sm6k"]
Sep 10 21:00:46.840405 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.840375    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-oauth-config\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.840558 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.840412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-config\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.840558 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.840432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-trusted-ca-bundle\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.840558 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.840462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzk7f\" (UniqueName: \"kubernetes.io/projected/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-kube-api-access-zzk7f\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.840558 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.840525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-oauth-serving-cert\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.840558 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.840544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-serving-cert\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.840558 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.840559    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-service-ca\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.941450 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.941413    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-oauth-serving-cert\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.941594 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.941455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-serving-cert\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.941594 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.941478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-service-ca\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.941594 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.941516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-oauth-config\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.941594 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.941552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-config\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.941594 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.941579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-trusted-ca-bundle\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.941828 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.941603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzk7f\" (UniqueName: \"kubernetes.io/projected/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-kube-api-access-zzk7f\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.942257 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.942226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-oauth-serving-cert\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.942362 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.942304    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-service-ca\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.942362 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.942327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-config\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.942552 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.942529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-trusted-ca-bundle\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.944099 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.944077    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-oauth-config\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.944148 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.944083    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-serving-cert\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:46.958146 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:46.958082    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzk7f\" (UniqueName: \"kubernetes.io/projected/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-kube-api-access-zzk7f\") pod \"console-58b4874f44-5sm6k\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") " pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:47.043507 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:47.043471    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:47.315159 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:47.315138    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58b4874f44-5sm6k"]
Sep 10 21:00:47.317805 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:00:47.317779    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede14eb3_2e94_4ae0_8d58_4f4b627e3f43.slice/crio-c8d80db09fa53801760069c400c24d8e0b73788244807ddf73e37c6db58173f8 WatchSource:0}: Error finding container c8d80db09fa53801760069c400c24d8e0b73788244807ddf73e37c6db58173f8: Status 404 returned error can't find the container with id c8d80db09fa53801760069c400c24d8e0b73788244807ddf73e37c6db58173f8
Sep 10 21:00:47.501253 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:47.501211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b4874f44-5sm6k" event={"ID":"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43","Type":"ContainerStarted","Data":"e865e771734ad3ae4a1e60e5f917fdd38248a0f9c582757c588a041f3a389e20"}
Sep 10 21:00:47.501444 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:47.501268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b4874f44-5sm6k" event={"ID":"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43","Type":"ContainerStarted","Data":"c8d80db09fa53801760069c400c24d8e0b73788244807ddf73e37c6db58173f8"}
Sep 10 21:00:47.501576 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:47.501458    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="kube-rbac-proxy" containerID="cri-o://e94e23c96b3132649ffad134c675ef219e3f6fe6e26b29b83ac9095afe1c9d71" gracePeriod=120
Sep 10 21:00:47.501576 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:47.501474    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="prom-label-proxy" containerID="cri-o://ee3bf62ce7c13251d84322174230b99aa616b34f6b98088142fd501de43b0f0a" gracePeriod=120
Sep 10 21:00:47.501576 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:47.501509    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="config-reloader" containerID="cri-o://085a47919657564f8a8a3dbeea059cfbda3b2f10bb79270141325fc1e8ec9f08" gracePeriod=120
Sep 10 21:00:47.501576 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:47.501520    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="kube-rbac-proxy-metric" containerID="cri-o://492f5934570d670164a4e5d23f77d44c08fb19aee995eecd104c7aa334da7c84" gracePeriod=120
Sep 10 21:00:47.501576 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:47.501456    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="alertmanager" containerID="cri-o://fa53573ae0be1016ce0ad3aaf696d24714a37599d2043964bc2084abd9b99db9" gracePeriod=120
Sep 10 21:00:47.501839 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:47.501472    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="kube-rbac-proxy-web" containerID="cri-o://e765e388a53c47581bd2edc173da9d189db46a6c4963c9445acf1e3c9428029c" gracePeriod=120
Sep 10 21:00:47.524894 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:47.524852    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58b4874f44-5sm6k" podStartSLOduration=1.524837689 podStartE2EDuration="1.524837689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:47.523770571 +0000 UTC m=+188.499430647" watchObservedRunningTime="2026-09-10 21:00:47.524837689 +0000 UTC m=+188.500497735"
Sep 10 21:00:48.508405 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.508374    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerID="ee3bf62ce7c13251d84322174230b99aa616b34f6b98088142fd501de43b0f0a" exitCode=0
Sep 10 21:00:48.508405 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.508401    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerID="492f5934570d670164a4e5d23f77d44c08fb19aee995eecd104c7aa334da7c84" exitCode=0
Sep 10 21:00:48.508405 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.508407    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerID="e94e23c96b3132649ffad134c675ef219e3f6fe6e26b29b83ac9095afe1c9d71" exitCode=0
Sep 10 21:00:48.508405 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.508415    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerID="085a47919657564f8a8a3dbeea059cfbda3b2f10bb79270141325fc1e8ec9f08" exitCode=0
Sep 10 21:00:48.509019 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.508422    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerID="fa53573ae0be1016ce0ad3aaf696d24714a37599d2043964bc2084abd9b99db9" exitCode=0
Sep 10 21:00:48.509019 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.508450    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e733ef7-73f2-40cd-bfeb-020645b2b3ee","Type":"ContainerDied","Data":"ee3bf62ce7c13251d84322174230b99aa616b34f6b98088142fd501de43b0f0a"}
Sep 10 21:00:48.509019 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.508536    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e733ef7-73f2-40cd-bfeb-020645b2b3ee","Type":"ContainerDied","Data":"492f5934570d670164a4e5d23f77d44c08fb19aee995eecd104c7aa334da7c84"}
Sep 10 21:00:48.509019 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.508585    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e733ef7-73f2-40cd-bfeb-020645b2b3ee","Type":"ContainerDied","Data":"e94e23c96b3132649ffad134c675ef219e3f6fe6e26b29b83ac9095afe1c9d71"}
Sep 10 21:00:48.509019 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.508626    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e733ef7-73f2-40cd-bfeb-020645b2b3ee","Type":"ContainerDied","Data":"085a47919657564f8a8a3dbeea059cfbda3b2f10bb79270141325fc1e8ec9f08"}
Sep 10 21:00:48.509019 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.508672    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e733ef7-73f2-40cd-bfeb-020645b2b3ee","Type":"ContainerDied","Data":"fa53573ae0be1016ce0ad3aaf696d24714a37599d2043964bc2084abd9b99db9"}
Sep 10 21:00:48.737358 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.737334    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:48.860301 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.860258    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-alertmanager-main-db\") pod \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") "
Sep 10 21:00:48.860476 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.860327    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-config-volume\") pod \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") "
Sep 10 21:00:48.860476 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.860353    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy\") pod \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") "
Sep 10 21:00:48.860476 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.860372    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-config-out\") pod \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") "
Sep 10 21:00:48.860476 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.860391    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-tls-assets\") pod \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") "
Sep 10 21:00:48.860476 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.860420    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") "
Sep 10 21:00:48.860762 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.860497    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz69f\" (UniqueName: \"kubernetes.io/projected/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-kube-api-access-cz69f\") pod \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") "
Sep 10 21:00:48.860762 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.860541    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-metrics-client-ca\") pod \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") "
Sep 10 21:00:48.860762 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.860576    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "8e733ef7-73f2-40cd-bfeb-020645b2b3ee" (UID: "8e733ef7-73f2-40cd-bfeb-020645b2b3ee"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:00:48.860762 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.860602    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-alertmanager-trusted-ca-bundle\") pod \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") "
Sep 10 21:00:48.860762 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.860628    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-web-config\") pod \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") "
Sep 10 21:00:48.860762 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.860661    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") "
Sep 10 21:00:48.860762 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.860693    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-main-tls\") pod \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") "
Sep 10 21:00:48.860762 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.860746    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-cluster-tls-config\") pod \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\" (UID: \"8e733ef7-73f2-40cd-bfeb-020645b2b3ee\") "
Sep 10 21:00:48.861225 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.860975    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-alertmanager-main-db\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:48.861225 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.861068    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "8e733ef7-73f2-40cd-bfeb-020645b2b3ee" (UID: "8e733ef7-73f2-40cd-bfeb-020645b2b3ee"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:48.861473 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.861450    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "8e733ef7-73f2-40cd-bfeb-020645b2b3ee" (UID: "8e733ef7-73f2-40cd-bfeb-020645b2b3ee"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:48.863310 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.863283    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-config-out" (OuterVolumeSpecName: "config-out") pod "8e733ef7-73f2-40cd-bfeb-020645b2b3ee" (UID: "8e733ef7-73f2-40cd-bfeb-020645b2b3ee"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:00:48.863677 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.863647    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "8e733ef7-73f2-40cd-bfeb-020645b2b3ee" (UID: "8e733ef7-73f2-40cd-bfeb-020645b2b3ee"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:48.864004 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.863959    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8e733ef7-73f2-40cd-bfeb-020645b2b3ee" (UID: "8e733ef7-73f2-40cd-bfeb-020645b2b3ee"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:00:48.864102 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.864066    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-kube-api-access-cz69f" (OuterVolumeSpecName: "kube-api-access-cz69f") pod "8e733ef7-73f2-40cd-bfeb-020645b2b3ee" (UID: "8e733ef7-73f2-40cd-bfeb-020645b2b3ee"). InnerVolumeSpecName "kube-api-access-cz69f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:00:48.864319 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.864291    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "8e733ef7-73f2-40cd-bfeb-020645b2b3ee" (UID: "8e733ef7-73f2-40cd-bfeb-020645b2b3ee"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:48.864490 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.864471    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-config-volume" (OuterVolumeSpecName: "config-volume") pod "8e733ef7-73f2-40cd-bfeb-020645b2b3ee" (UID: "8e733ef7-73f2-40cd-bfeb-020645b2b3ee"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:48.864552 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.864497    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "8e733ef7-73f2-40cd-bfeb-020645b2b3ee" (UID: "8e733ef7-73f2-40cd-bfeb-020645b2b3ee"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:48.864834 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.864815    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "8e733ef7-73f2-40cd-bfeb-020645b2b3ee" (UID: "8e733ef7-73f2-40cd-bfeb-020645b2b3ee"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:48.867541 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.867520    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "8e733ef7-73f2-40cd-bfeb-020645b2b3ee" (UID: "8e733ef7-73f2-40cd-bfeb-020645b2b3ee"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:48.873948 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.873924    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-web-config" (OuterVolumeSpecName: "web-config") pod "8e733ef7-73f2-40cd-bfeb-020645b2b3ee" (UID: "8e733ef7-73f2-40cd-bfeb-020645b2b3ee"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:48.961555 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.961525    2581 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-config-volume\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:48.961555 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.961552    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:48.961699 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.961561    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-config-out\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:48.961699 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.961570    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-tls-assets\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:48.961699 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.961577    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:48.961699 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.961587    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz69f\" (UniqueName: \"kubernetes.io/projected/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-kube-api-access-cz69f\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:48.961699 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.961595    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-metrics-client-ca\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:48.961699 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.961604    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:48.961699 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.961612    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-web-config\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:48.961699 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.961620    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:48.961699 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.961628    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-secret-alertmanager-main-tls\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:48.961699 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:48.961636    2581 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8e733ef7-73f2-40cd-bfeb-020645b2b3ee-cluster-tls-config\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:00:49.515538 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.515497    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerID="e765e388a53c47581bd2edc173da9d189db46a6c4963c9445acf1e3c9428029c" exitCode=0
Sep 10 21:00:49.515961 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.515571    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e733ef7-73f2-40cd-bfeb-020645b2b3ee","Type":"ContainerDied","Data":"e765e388a53c47581bd2edc173da9d189db46a6c4963c9445acf1e3c9428029c"}
Sep 10 21:00:49.515961 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.515582    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.515961 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.515703    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e733ef7-73f2-40cd-bfeb-020645b2b3ee","Type":"ContainerDied","Data":"1e2320a2b40b94b172d59aa106fd0010dbed97959af3c0e893553375d8024f65"}
Sep 10 21:00:49.515961 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.515752    2581 scope.go:117] "RemoveContainer" containerID="ee3bf62ce7c13251d84322174230b99aa616b34f6b98088142fd501de43b0f0a"
Sep 10 21:00:49.524227 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.524181    2581 scope.go:117] "RemoveContainer" containerID="492f5934570d670164a4e5d23f77d44c08fb19aee995eecd104c7aa334da7c84"
Sep 10 21:00:49.531293 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.531277    2581 scope.go:117] "RemoveContainer" containerID="e94e23c96b3132649ffad134c675ef219e3f6fe6e26b29b83ac9095afe1c9d71"
Sep 10 21:00:49.538554 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.538538    2581 scope.go:117] "RemoveContainer" containerID="e765e388a53c47581bd2edc173da9d189db46a6c4963c9445acf1e3c9428029c"
Sep 10 21:00:49.541345 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.541323    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 21:00:49.546709 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.546694    2581 scope.go:117] "RemoveContainer" containerID="085a47919657564f8a8a3dbeea059cfbda3b2f10bb79270141325fc1e8ec9f08"
Sep 10 21:00:49.546864 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.546843    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 21:00:49.553226 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.553193    2581 scope.go:117] "RemoveContainer" containerID="fa53573ae0be1016ce0ad3aaf696d24714a37599d2043964bc2084abd9b99db9"
Sep 10 21:00:49.559767 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.559754    2581 scope.go:117] "RemoveContainer" containerID="fbf50c561d0fccf7dc23589f34dcc8c8c615f168077a5757b2be442c945564f4"
Sep 10 21:00:49.566125 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.566109    2581 scope.go:117] "RemoveContainer" containerID="ee3bf62ce7c13251d84322174230b99aa616b34f6b98088142fd501de43b0f0a"
Sep 10 21:00:49.566432 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:00:49.566408    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee3bf62ce7c13251d84322174230b99aa616b34f6b98088142fd501de43b0f0a\": container with ID starting with ee3bf62ce7c13251d84322174230b99aa616b34f6b98088142fd501de43b0f0a not found: ID does not exist" containerID="ee3bf62ce7c13251d84322174230b99aa616b34f6b98088142fd501de43b0f0a"
Sep 10 21:00:49.566527 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.566438    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee3bf62ce7c13251d84322174230b99aa616b34f6b98088142fd501de43b0f0a"} err="failed to get container status \"ee3bf62ce7c13251d84322174230b99aa616b34f6b98088142fd501de43b0f0a\": rpc error: code = NotFound desc = could not find container \"ee3bf62ce7c13251d84322174230b99aa616b34f6b98088142fd501de43b0f0a\": container with ID starting with ee3bf62ce7c13251d84322174230b99aa616b34f6b98088142fd501de43b0f0a not found: ID does not exist"
Sep 10 21:00:49.566527 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.566456    2581 scope.go:117] "RemoveContainer" containerID="492f5934570d670164a4e5d23f77d44c08fb19aee995eecd104c7aa334da7c84"
Sep 10 21:00:49.566867 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:00:49.566847    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492f5934570d670164a4e5d23f77d44c08fb19aee995eecd104c7aa334da7c84\": container with ID starting with 492f5934570d670164a4e5d23f77d44c08fb19aee995eecd104c7aa334da7c84 not found: ID does not exist" containerID="492f5934570d670164a4e5d23f77d44c08fb19aee995eecd104c7aa334da7c84"
Sep 10 21:00:49.566963 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.566870    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492f5934570d670164a4e5d23f77d44c08fb19aee995eecd104c7aa334da7c84"} err="failed to get container status \"492f5934570d670164a4e5d23f77d44c08fb19aee995eecd104c7aa334da7c84\": rpc error: code = NotFound desc = could not find container \"492f5934570d670164a4e5d23f77d44c08fb19aee995eecd104c7aa334da7c84\": container with ID starting with 492f5934570d670164a4e5d23f77d44c08fb19aee995eecd104c7aa334da7c84 not found: ID does not exist"
Sep 10 21:00:49.566963 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.566885    2581 scope.go:117] "RemoveContainer" containerID="e94e23c96b3132649ffad134c675ef219e3f6fe6e26b29b83ac9095afe1c9d71"
Sep 10 21:00:49.567165 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:00:49.567145    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94e23c96b3132649ffad134c675ef219e3f6fe6e26b29b83ac9095afe1c9d71\": container with ID starting with e94e23c96b3132649ffad134c675ef219e3f6fe6e26b29b83ac9095afe1c9d71 not found: ID does not exist" containerID="e94e23c96b3132649ffad134c675ef219e3f6fe6e26b29b83ac9095afe1c9d71"
Sep 10 21:00:49.567230 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.567170    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94e23c96b3132649ffad134c675ef219e3f6fe6e26b29b83ac9095afe1c9d71"} err="failed to get container status \"e94e23c96b3132649ffad134c675ef219e3f6fe6e26b29b83ac9095afe1c9d71\": rpc error: code = NotFound desc = could not find container \"e94e23c96b3132649ffad134c675ef219e3f6fe6e26b29b83ac9095afe1c9d71\": container with ID starting with e94e23c96b3132649ffad134c675ef219e3f6fe6e26b29b83ac9095afe1c9d71 not found: ID does not exist"
Sep 10 21:00:49.567230 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.567190    2581 scope.go:117] "RemoveContainer" containerID="e765e388a53c47581bd2edc173da9d189db46a6c4963c9445acf1e3c9428029c"
Sep 10 21:00:49.567501 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:00:49.567481    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e765e388a53c47581bd2edc173da9d189db46a6c4963c9445acf1e3c9428029c\": container with ID starting with e765e388a53c47581bd2edc173da9d189db46a6c4963c9445acf1e3c9428029c not found: ID does not exist" containerID="e765e388a53c47581bd2edc173da9d189db46a6c4963c9445acf1e3c9428029c"
Sep 10 21:00:49.567565 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.567504    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e765e388a53c47581bd2edc173da9d189db46a6c4963c9445acf1e3c9428029c"} err="failed to get container status \"e765e388a53c47581bd2edc173da9d189db46a6c4963c9445acf1e3c9428029c\": rpc error: code = NotFound desc = could not find container \"e765e388a53c47581bd2edc173da9d189db46a6c4963c9445acf1e3c9428029c\": container with ID starting with e765e388a53c47581bd2edc173da9d189db46a6c4963c9445acf1e3c9428029c not found: ID does not exist"
Sep 10 21:00:49.567565 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.567516    2581 scope.go:117] "RemoveContainer" containerID="085a47919657564f8a8a3dbeea059cfbda3b2f10bb79270141325fc1e8ec9f08"
Sep 10 21:00:49.567765 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:00:49.567750    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"085a47919657564f8a8a3dbeea059cfbda3b2f10bb79270141325fc1e8ec9f08\": container with ID starting with 085a47919657564f8a8a3dbeea059cfbda3b2f10bb79270141325fc1e8ec9f08 not found: ID does not exist" containerID="085a47919657564f8a8a3dbeea059cfbda3b2f10bb79270141325fc1e8ec9f08"
Sep 10 21:00:49.567825 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.567766    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085a47919657564f8a8a3dbeea059cfbda3b2f10bb79270141325fc1e8ec9f08"} err="failed to get container status \"085a47919657564f8a8a3dbeea059cfbda3b2f10bb79270141325fc1e8ec9f08\": rpc error: code = NotFound desc = could not find container \"085a47919657564f8a8a3dbeea059cfbda3b2f10bb79270141325fc1e8ec9f08\": container with ID starting with 085a47919657564f8a8a3dbeea059cfbda3b2f10bb79270141325fc1e8ec9f08 not found: ID does not exist"
Sep 10 21:00:49.567825 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.567778    2581 scope.go:117] "RemoveContainer" containerID="fa53573ae0be1016ce0ad3aaf696d24714a37599d2043964bc2084abd9b99db9"
Sep 10 21:00:49.568008 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:00:49.567991    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa53573ae0be1016ce0ad3aaf696d24714a37599d2043964bc2084abd9b99db9\": container with ID starting with fa53573ae0be1016ce0ad3aaf696d24714a37599d2043964bc2084abd9b99db9 not found: ID does not exist" containerID="fa53573ae0be1016ce0ad3aaf696d24714a37599d2043964bc2084abd9b99db9"
Sep 10 21:00:49.568072 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.568011    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa53573ae0be1016ce0ad3aaf696d24714a37599d2043964bc2084abd9b99db9"} err="failed to get container status \"fa53573ae0be1016ce0ad3aaf696d24714a37599d2043964bc2084abd9b99db9\": rpc error: code = NotFound desc = could not find container \"fa53573ae0be1016ce0ad3aaf696d24714a37599d2043964bc2084abd9b99db9\": container with ID starting with fa53573ae0be1016ce0ad3aaf696d24714a37599d2043964bc2084abd9b99db9 not found: ID does not exist"
Sep 10 21:00:49.568072 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.568023    2581 scope.go:117] "RemoveContainer" containerID="fbf50c561d0fccf7dc23589f34dcc8c8c615f168077a5757b2be442c945564f4"
Sep 10 21:00:49.568315 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:00:49.568290    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbf50c561d0fccf7dc23589f34dcc8c8c615f168077a5757b2be442c945564f4\": container with ID starting with fbf50c561d0fccf7dc23589f34dcc8c8c615f168077a5757b2be442c945564f4 not found: ID does not exist" containerID="fbf50c561d0fccf7dc23589f34dcc8c8c615f168077a5757b2be442c945564f4"
Sep 10 21:00:49.568436 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.568320    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbf50c561d0fccf7dc23589f34dcc8c8c615f168077a5757b2be442c945564f4"} err="failed to get container status \"fbf50c561d0fccf7dc23589f34dcc8c8c615f168077a5757b2be442c945564f4\": rpc error: code = NotFound desc = could not find container \"fbf50c561d0fccf7dc23589f34dcc8c8c615f168077a5757b2be442c945564f4\": container with ID starting with fbf50c561d0fccf7dc23589f34dcc8c8c615f168077a5757b2be442c945564f4 not found: ID does not exist"
Sep 10 21:00:49.572601 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.572582    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" path="/var/lib/kubelet/pods/8e733ef7-73f2-40cd-bfeb-020645b2b3ee/volumes"
Sep 10 21:00:49.576579 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576561    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 21:00:49.576835 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576823    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="alertmanager"
Sep 10 21:00:49.576873 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576838    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="alertmanager"
Sep 10 21:00:49.576873 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576854    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="kube-rbac-proxy-metric"
Sep 10 21:00:49.576873 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576863    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="kube-rbac-proxy-metric"
Sep 10 21:00:49.576873 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576871    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="prom-label-proxy"
Sep 10 21:00:49.576988 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576876    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="prom-label-proxy"
Sep 10 21:00:49.576988 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576887    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="kube-rbac-proxy-web"
Sep 10 21:00:49.576988 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576892    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="kube-rbac-proxy-web"
Sep 10 21:00:49.576988 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576900    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="config-reloader"
Sep 10 21:00:49.576988 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576905    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="config-reloader"
Sep 10 21:00:49.576988 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576912    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="init-config-reloader"
Sep 10 21:00:49.576988 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576917    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="init-config-reloader"
Sep 10 21:00:49.576988 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576926    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="kube-rbac-proxy"
Sep 10 21:00:49.576988 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576935    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="kube-rbac-proxy"
Sep 10 21:00:49.576988 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576983    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="kube-rbac-proxy-web"
Sep 10 21:00:49.576988 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576992    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="prom-label-proxy"
Sep 10 21:00:49.577301 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.576999    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="kube-rbac-proxy-metric"
Sep 10 21:00:49.577301 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.577007    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="config-reloader"
Sep 10 21:00:49.577301 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.577016    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="kube-rbac-proxy"
Sep 10 21:00:49.577301 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.577023    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e733ef7-73f2-40cd-bfeb-020645b2b3ee" containerName="alertmanager"
Sep 10 21:00:49.581781 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.581766    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.584019 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.584000    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 21:00:49.584019 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.584015    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 21:00:49.584259 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.584245    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 21:00:49.584340 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.584315    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 21:00:49.584340 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.584316    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4rpfc\""
Sep 10 21:00:49.584445 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.584378    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 21:00:49.584502 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.584459    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 21:00:49.584639 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.584623    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 21:00:49.584710 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.584680    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 21:00:49.591544 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.591524    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 21:00:49.600740 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.600664    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 21:00:49.666712 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.666679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-config-volume\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.666712 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.666715    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c83819f-940a-42b0-95a8-92a8d8359de7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.666909 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.666732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-web-config\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.666909 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.666825    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.666909 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.666846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c83819f-940a-42b0-95a8-92a8d8359de7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.666909 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.666863    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.667059 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.666931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.667059 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.666958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c83819f-940a-42b0-95a8-92a8d8359de7-config-out\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.667059 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.666974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.667059 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.666989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c83819f-940a-42b0-95a8-92a8d8359de7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.667059 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.667024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.667059 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.667040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vpsk\" (UniqueName: \"kubernetes.io/projected/2c83819f-940a-42b0-95a8-92a8d8359de7-kube-api-access-5vpsk\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.667315 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.667114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c83819f-940a-42b0-95a8-92a8d8359de7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.768220 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.768125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c83819f-940a-42b0-95a8-92a8d8359de7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.768220 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.768157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-web-config\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.768220 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.768188    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.768431 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.768392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c83819f-940a-42b0-95a8-92a8d8359de7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.768431 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.768425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.768602 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.768573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.768791 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.768772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c83819f-940a-42b0-95a8-92a8d8359de7-config-out\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.768909 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.768892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.769018 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.768998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c83819f-940a-42b0-95a8-92a8d8359de7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.769121 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.769107    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.769254 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.769239    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vpsk\" (UniqueName: \"kubernetes.io/projected/2c83819f-940a-42b0-95a8-92a8d8359de7-kube-api-access-5vpsk\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.769385 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.769371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c83819f-940a-42b0-95a8-92a8d8359de7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.769535 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.769519    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-config-volume\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.769754 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.769728    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c83819f-940a-42b0-95a8-92a8d8359de7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.771222 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.771185    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c83819f-940a-42b0-95a8-92a8d8359de7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.772010 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.771349    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-web-config\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.772010 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.771358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.772010 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.771538    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.772010 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.769395    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c83819f-940a-42b0-95a8-92a8d8359de7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.772010 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.771974    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c83819f-940a-42b0-95a8-92a8d8359de7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.772315 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.772133    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.772315 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.772271    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c83819f-940a-42b0-95a8-92a8d8359de7-config-out\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.772525 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.772504    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-config-volume\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.773685 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.773660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.773786 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.773694    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c83819f-940a-42b0-95a8-92a8d8359de7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.779595 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.779571    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vpsk\" (UniqueName: \"kubernetes.io/projected/2c83819f-940a-42b0-95a8-92a8d8359de7-kube-api-access-5vpsk\") pod \"alertmanager-main-0\" (UID: \"2c83819f-940a-42b0-95a8-92a8d8359de7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:49.891809 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:49.891779    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 21:00:50.177216 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:50.177180    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 21:00:50.179667 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:00:50.179637    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c83819f_940a_42b0_95a8_92a8d8359de7.slice/crio-e54de77f7caee8f711456d0d0f4eb46c2546ad090501ac2d59b99be0b4a53218 WatchSource:0}: Error finding container e54de77f7caee8f711456d0d0f4eb46c2546ad090501ac2d59b99be0b4a53218: Status 404 returned error can't find the container with id e54de77f7caee8f711456d0d0f4eb46c2546ad090501ac2d59b99be0b4a53218
Sep 10 21:00:50.525917 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:50.525839    2581 generic.go:356] "Generic (PLEG): container finished" podID="2c83819f-940a-42b0-95a8-92a8d8359de7" containerID="50a09942f43a1572f053a30cc9ae778cc06347bde01f7fef089daa4030e80330" exitCode=0
Sep 10 21:00:50.526349 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:50.525929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c83819f-940a-42b0-95a8-92a8d8359de7","Type":"ContainerDied","Data":"50a09942f43a1572f053a30cc9ae778cc06347bde01f7fef089daa4030e80330"}
Sep 10 21:00:50.526349 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:50.526037    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c83819f-940a-42b0-95a8-92a8d8359de7","Type":"ContainerStarted","Data":"e54de77f7caee8f711456d0d0f4eb46c2546ad090501ac2d59b99be0b4a53218"}
Sep 10 21:00:51.533514 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:51.533482    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c83819f-940a-42b0-95a8-92a8d8359de7","Type":"ContainerStarted","Data":"e54cd8011471d1908919dd81312ce6ea8afc80ffa34a28f931379a3c333ce2e6"}
Sep 10 21:00:51.533886 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:51.533547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c83819f-940a-42b0-95a8-92a8d8359de7","Type":"ContainerStarted","Data":"2da8d3e2980a1160f2ace69e6a17217706ab3a5e271a1b9ef0a5d78f392a1207"}
Sep 10 21:00:51.533886 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:51.533575    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c83819f-940a-42b0-95a8-92a8d8359de7","Type":"ContainerStarted","Data":"45197a8457c0c26bf92139cc37b6a30f4f904f4f91f64d26d929fb74e00324ff"}
Sep 10 21:00:51.533886 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:51.533600    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c83819f-940a-42b0-95a8-92a8d8359de7","Type":"ContainerStarted","Data":"d08da10ebf24ddb6393ee4d71e024f972b1e9e5dd49dd27ba5720540156996ad"}
Sep 10 21:00:51.533886 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:51.533634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c83819f-940a-42b0-95a8-92a8d8359de7","Type":"ContainerStarted","Data":"1013f4526899bec0cf7a7ea36a729590faa55a08bb44c94772f1ade0925e7ffa"}
Sep 10 21:00:51.533886 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:51.533668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c83819f-940a-42b0-95a8-92a8d8359de7","Type":"ContainerStarted","Data":"8e1dda7da3a9af5cba8d5eb81107160dedb731b2bd6377d0b85caf2c1654e732"}
Sep 10 21:00:51.564680 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:51.564622    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.54959011 podStartE2EDuration="2.564599554s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.009444ms" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:51.563369502 +0000 UTC m=+192.539029549" watchObservedRunningTime="2026-09-10 21:00:51.564599554 +0000 UTC m=+192.540259603"
Sep 10 21:00:52.703193 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.703158    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:00:52.707136 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.707116    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.710370 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.710353    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 21:00:52.710678 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.710646    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 21:00:52.710678 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.710656    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 21:00:52.710878 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.710709    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-h4wm8\""
Sep 10 21:00:52.710878 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.710721    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 21:00:52.710878 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.710656    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-90637mv87vj6u\""
Sep 10 21:00:52.710878 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.710724    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 21:00:52.710878 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.710666    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 21:00:52.711227 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.711153    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 21:00:52.711227 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.711190    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 21:00:52.712337 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.712317    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 21:00:52.712446 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.712342    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 21:00:52.712446 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.712352    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 21:00:52.712446 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.712363    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 21:00:52.717411 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.717386    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 21:00:52.719657 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.719633    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 21:00:52.724196 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.724171    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:00:52.791954 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.791920    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792103 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.791960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792103 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.791979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792103 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.791999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792103 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.792045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-web-config\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792103 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.792084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fee12dc3-35c4-406c-a0cd-0df23feea5f3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792337 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.792106    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792337 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.792134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792337 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.792160    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792337 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.792174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792337 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.792191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv97f\" (UniqueName: \"kubernetes.io/projected/fee12dc3-35c4-406c-a0cd-0df23feea5f3-kube-api-access-tv97f\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792337 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.792238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792337 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.792269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792337 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.792287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fee12dc3-35c4-406c-a0cd-0df23feea5f3-config-out\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792337 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.792315    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792611 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.792345    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-config\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792611 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.792362    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792611 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.792383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792611 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.792405    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fee12dc3-35c4-406c-a0cd-0df23feea5f3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.792611 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.792422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893029 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.892992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893029 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893225 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893225 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893163    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-web-config\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893309 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fee12dc3-35c4-406c-a0cd-0df23feea5f3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893309 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893254    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893309 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893517 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893517 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893517 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv97f\" (UniqueName: \"kubernetes.io/projected/fee12dc3-35c4-406c-a0cd-0df23feea5f3-kube-api-access-tv97f\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893517 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893517 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893517 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fee12dc3-35c4-406c-a0cd-0df23feea5f3-config-out\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893517 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893493    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893882 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-config\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893882 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893882 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893882 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fee12dc3-35c4-406c-a0cd-0df23feea5f3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893882 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893643    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.893882 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893693    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fee12dc3-35c4-406c-a0cd-0df23feea5f3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.894222 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893982    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.894287 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.894238    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.894287 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.893699    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.894695 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.894668    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.895479 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.895386    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.895881 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.895858    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.896003 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.895981    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.896143 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.896122    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fee12dc3-35c4-406c-a0cd-0df23feea5f3-config-out\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.896235 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.896195    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.896307 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.896238    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.897250 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.896863    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-web-config\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.897250 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.896887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.897506 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.897484    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fee12dc3-35c4-406c-a0cd-0df23feea5f3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.898913 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.898692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fee12dc3-35c4-406c-a0cd-0df23feea5f3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.898913 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.898857    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.898913 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.898864    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.899264 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.899242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.899380 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.899365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.899522 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.899504    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fee12dc3-35c4-406c-a0cd-0df23feea5f3-config\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.904280 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:52.904258    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv97f\" (UniqueName: \"kubernetes.io/projected/fee12dc3-35c4-406c-a0cd-0df23feea5f3-kube-api-access-tv97f\") pod \"prometheus-k8s-0\" (UID: \"fee12dc3-35c4-406c-a0cd-0df23feea5f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:53.018185 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:53.018114    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:53.307529 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:53.307506    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:00:53.309088 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:00:53.309051    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee12dc3_35c4_406c_a0cd_0df23feea5f3.slice/crio-7a205fbf0b676e5ccd541d6a5916c5a9b539f8bb601e6c33b03710878caf4e0d WatchSource:0}: Error finding container 7a205fbf0b676e5ccd541d6a5916c5a9b539f8bb601e6c33b03710878caf4e0d: Status 404 returned error can't find the container with id 7a205fbf0b676e5ccd541d6a5916c5a9b539f8bb601e6c33b03710878caf4e0d
Sep 10 21:00:53.543880 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:53.543845    2581 generic.go:356] "Generic (PLEG): container finished" podID="fee12dc3-35c4-406c-a0cd-0df23feea5f3" containerID="288c1309d5216794de733f2782941ce49c24b021d650740e8584338fffc336ec" exitCode=0
Sep 10 21:00:53.544058 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:53.543936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fee12dc3-35c4-406c-a0cd-0df23feea5f3","Type":"ContainerDied","Data":"288c1309d5216794de733f2782941ce49c24b021d650740e8584338fffc336ec"}
Sep 10 21:00:53.544058 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:53.544024    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fee12dc3-35c4-406c-a0cd-0df23feea5f3","Type":"ContainerStarted","Data":"7a205fbf0b676e5ccd541d6a5916c5a9b539f8bb601e6c33b03710878caf4e0d"}
Sep 10 21:00:57.043723 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:57.043685    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:57.044134 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:57.043756    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:57.048356 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:57.048334    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:57.566404 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:57.566367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fee12dc3-35c4-406c-a0cd-0df23feea5f3","Type":"ContainerStarted","Data":"47e64dcd40505c310a26b810a9de7bcdd321e21032f636041fc28dad72d6e0e0"}
Sep 10 21:00:57.566596 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:57.566429    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fee12dc3-35c4-406c-a0cd-0df23feea5f3","Type":"ContainerStarted","Data":"f67b11cb7b723962490fb42379e6dd4cf616070a93403235417fe184e98ab527"}
Sep 10 21:00:57.566596 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:57.566457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fee12dc3-35c4-406c-a0cd-0df23feea5f3","Type":"ContainerStarted","Data":"2d376fc254a710ab108805ac60d7ae7a99454e9733f357f3f69fd54af3678287"}
Sep 10 21:00:57.566596 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:57.566481    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fee12dc3-35c4-406c-a0cd-0df23feea5f3","Type":"ContainerStarted","Data":"cc8cd182bc791b0acc57b86260c32fa6e2d13d81968729f479ab1211635699d1"}
Sep 10 21:00:57.566596 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:57.566503    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fee12dc3-35c4-406c-a0cd-0df23feea5f3","Type":"ContainerStarted","Data":"edd05d2f6b5393dcd8021f1d722cb66865fd31c94fcbcc2b1b8bed6cf51550c2"}
Sep 10 21:00:57.566596 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:57.566529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fee12dc3-35c4-406c-a0cd-0df23feea5f3","Type":"ContainerStarted","Data":"57489c71b4cd4c412ec3629de358e9a6c7716289916abc0b8a9c70aa9ea17a24"}
Sep 10 21:00:57.572604 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:57.572580    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:00:57.605389 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:57.601863    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.491217296 podStartE2EDuration="5.601840454s" totalImagesPullingTime="3.094610028s" totalInitContainerRuntime="16.01313ms" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:57.597541373 +0000 UTC m=+198.573201420" watchObservedRunningTime="2026-09-10 21:00:57.601840454 +0000 UTC m=+198.577500502"
Sep 10 21:00:58.018929 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:00:58.018851    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:01:53.019265 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:01:53.019231    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:01:53.037656 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:01:53.037631    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:01:53.807034 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:01:53.807008    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:02:24.397285 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:24.397249    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hsg2x"]
Sep 10 21:02:24.400453 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:24.400438    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hsg2x"
Sep 10 21:02:24.402578 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:24.402556    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 21:02:24.403217 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:24.403189    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 21:02:24.403308 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:24.403282    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-btfv6\""
Sep 10 21:02:24.411354 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:24.411334    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hsg2x"]
Sep 10 21:02:24.436442 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:24.436417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7t7k\" (UniqueName: \"kubernetes.io/projected/2dbcfdbd-a0e1-4833-ad84-2d99a6d3f04b-kube-api-access-p7t7k\") pod \"openshift-lws-operator-fd8ccff4c-hsg2x\" (UID: \"2dbcfdbd-a0e1-4833-ad84-2d99a6d3f04b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hsg2x"
Sep 10 21:02:24.436535 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:24.436464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dbcfdbd-a0e1-4833-ad84-2d99a6d3f04b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hsg2x\" (UID: \"2dbcfdbd-a0e1-4833-ad84-2d99a6d3f04b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hsg2x"
Sep 10 21:02:24.537261 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:24.537225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7t7k\" (UniqueName: \"kubernetes.io/projected/2dbcfdbd-a0e1-4833-ad84-2d99a6d3f04b-kube-api-access-p7t7k\") pod \"openshift-lws-operator-fd8ccff4c-hsg2x\" (UID: \"2dbcfdbd-a0e1-4833-ad84-2d99a6d3f04b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hsg2x"
Sep 10 21:02:24.537426 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:24.537299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dbcfdbd-a0e1-4833-ad84-2d99a6d3f04b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hsg2x\" (UID: \"2dbcfdbd-a0e1-4833-ad84-2d99a6d3f04b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hsg2x"
Sep 10 21:02:24.537767 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:24.537745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dbcfdbd-a0e1-4833-ad84-2d99a6d3f04b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hsg2x\" (UID: \"2dbcfdbd-a0e1-4833-ad84-2d99a6d3f04b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hsg2x"
Sep 10 21:02:24.547055 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:24.547029    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7t7k\" (UniqueName: \"kubernetes.io/projected/2dbcfdbd-a0e1-4833-ad84-2d99a6d3f04b-kube-api-access-p7t7k\") pod \"openshift-lws-operator-fd8ccff4c-hsg2x\" (UID: \"2dbcfdbd-a0e1-4833-ad84-2d99a6d3f04b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hsg2x"
Sep 10 21:02:24.709563 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:24.709488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hsg2x"
Sep 10 21:02:24.975551 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:24.975526    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hsg2x"]
Sep 10 21:02:24.978217 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:02:24.978176    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dbcfdbd_a0e1_4833_ad84_2d99a6d3f04b.slice/crio-a9cd46bc7639062a33886238a0db4067aadd9f44141e25308d5ebd426e208e7d WatchSource:0}: Error finding container a9cd46bc7639062a33886238a0db4067aadd9f44141e25308d5ebd426e208e7d: Status 404 returned error can't find the container with id a9cd46bc7639062a33886238a0db4067aadd9f44141e25308d5ebd426e208e7d
Sep 10 21:02:25.918166 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:25.918123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hsg2x" event={"ID":"2dbcfdbd-a0e1-4833-ad84-2d99a6d3f04b","Type":"ContainerStarted","Data":"a9cd46bc7639062a33886238a0db4067aadd9f44141e25308d5ebd426e208e7d"}
Sep 10 21:02:27.929383 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:27.929349    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hsg2x" event={"ID":"2dbcfdbd-a0e1-4833-ad84-2d99a6d3f04b","Type":"ContainerStarted","Data":"f8830ef922e9eb6d9407195f94554f14e42fc3cfbd2812d0620890a33b177a8b"}
Sep 10 21:02:27.947743 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:27.947694    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hsg2x" podStartSLOduration=1.452607485 podStartE2EDuration="3.947679352s" totalImagesPullingTime="2.495071867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:27.947615809 +0000 UTC m=+288.923275852" watchObservedRunningTime="2026-09-10 21:02:27.947679352 +0000 UTC m=+288.923339399"
Sep 10 21:02:39.464773 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:39.464739    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 21:02:44.170193 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.170160    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"]
Sep 10 21:02:44.176465 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.176449    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:44.178643 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.178616    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 21:02:44.178770 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.178660    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 21:02:44.178915 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.178902    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 21:02:44.179397 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.179378    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-t9kg8\""
Sep 10 21:02:44.179397 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.179396    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 21:02:44.189772 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.189743    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"]
Sep 10 21:02:44.306632 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.306604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1863177c-3d62-433a-b69d-b2122d699c18-manifests\") pod \"opendatahub-operator-controller-manager-6d4c889df7-4zq9w\" (UID: \"1863177c-3d62-433a-b69d-b2122d699c18\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:44.306632 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.306635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1863177c-3d62-433a-b69d-b2122d699c18-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d4c889df7-4zq9w\" (UID: \"1863177c-3d62-433a-b69d-b2122d699c18\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:44.306839 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.306679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1863177c-3d62-433a-b69d-b2122d699c18-tmp\") pod \"opendatahub-operator-controller-manager-6d4c889df7-4zq9w\" (UID: \"1863177c-3d62-433a-b69d-b2122d699c18\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:44.306839 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.306719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1863177c-3d62-433a-b69d-b2122d699c18-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d4c889df7-4zq9w\" (UID: \"1863177c-3d62-433a-b69d-b2122d699c18\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:44.306839 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.306742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp6wx\" (UniqueName: \"kubernetes.io/projected/1863177c-3d62-433a-b69d-b2122d699c18-kube-api-access-kp6wx\") pod \"opendatahub-operator-controller-manager-6d4c889df7-4zq9w\" (UID: \"1863177c-3d62-433a-b69d-b2122d699c18\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:44.407557 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.407522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1863177c-3d62-433a-b69d-b2122d699c18-manifests\") pod \"opendatahub-operator-controller-manager-6d4c889df7-4zq9w\" (UID: \"1863177c-3d62-433a-b69d-b2122d699c18\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:44.407557 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.407555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1863177c-3d62-433a-b69d-b2122d699c18-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d4c889df7-4zq9w\" (UID: \"1863177c-3d62-433a-b69d-b2122d699c18\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:44.407747 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.407589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1863177c-3d62-433a-b69d-b2122d699c18-tmp\") pod \"opendatahub-operator-controller-manager-6d4c889df7-4zq9w\" (UID: \"1863177c-3d62-433a-b69d-b2122d699c18\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:44.407747 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.407707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1863177c-3d62-433a-b69d-b2122d699c18-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d4c889df7-4zq9w\" (UID: \"1863177c-3d62-433a-b69d-b2122d699c18\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:44.407822 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.407757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp6wx\" (UniqueName: \"kubernetes.io/projected/1863177c-3d62-433a-b69d-b2122d699c18-kube-api-access-kp6wx\") pod \"opendatahub-operator-controller-manager-6d4c889df7-4zq9w\" (UID: \"1863177c-3d62-433a-b69d-b2122d699c18\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:44.408079 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.408050    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1863177c-3d62-433a-b69d-b2122d699c18-manifests\") pod \"opendatahub-operator-controller-manager-6d4c889df7-4zq9w\" (UID: \"1863177c-3d62-433a-b69d-b2122d699c18\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:44.409966 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.409943    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1863177c-3d62-433a-b69d-b2122d699c18-tmp\") pod \"opendatahub-operator-controller-manager-6d4c889df7-4zq9w\" (UID: \"1863177c-3d62-433a-b69d-b2122d699c18\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:44.410072 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.410058    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1863177c-3d62-433a-b69d-b2122d699c18-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d4c889df7-4zq9w\" (UID: \"1863177c-3d62-433a-b69d-b2122d699c18\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:44.410122 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.410066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1863177c-3d62-433a-b69d-b2122d699c18-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d4c889df7-4zq9w\" (UID: \"1863177c-3d62-433a-b69d-b2122d699c18\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:44.416194 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.416177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp6wx\" (UniqueName: \"kubernetes.io/projected/1863177c-3d62-433a-b69d-b2122d699c18-kube-api-access-kp6wx\") pod \"opendatahub-operator-controller-manager-6d4c889df7-4zq9w\" (UID: \"1863177c-3d62-433a-b69d-b2122d699c18\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:44.487295 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.487230    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:44.766488 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.766464    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"]
Sep 10 21:02:44.772035 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:02:44.772008    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1863177c_3d62_433a_b69d_b2122d699c18.slice/crio-f2601400368247feceddf264b67c6229ad2a4d7075ff3bfb6ee98f7b04f11ff6 WatchSource:0}: Error finding container f2601400368247feceddf264b67c6229ad2a4d7075ff3bfb6ee98f7b04f11ff6: Status 404 returned error can't find the container with id f2601400368247feceddf264b67c6229ad2a4d7075ff3bfb6ee98f7b04f11ff6
Sep 10 21:02:44.773963 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:44.773946    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:02:45.001612 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:45.001572    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w" event={"ID":"1863177c-3d62-433a-b69d-b2122d699c18","Type":"ContainerStarted","Data":"f2601400368247feceddf264b67c6229ad2a4d7075ff3bfb6ee98f7b04f11ff6"}
Sep 10 21:02:48.018304 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:48.018222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w" event={"ID":"1863177c-3d62-433a-b69d-b2122d699c18","Type":"ContainerStarted","Data":"345428ff1a7de884aaae1827955f4d6ead964252b308b325c27e1beb8463cc52"}
Sep 10 21:02:49.024308 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:49.024220    2581 generic.go:356] "Generic (PLEG): container finished" podID="1863177c-3d62-433a-b69d-b2122d699c18" containerID="345428ff1a7de884aaae1827955f4d6ead964252b308b325c27e1beb8463cc52" exitCode=0
Sep 10 21:02:49.024679 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:49.024310    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w" event={"ID":"1863177c-3d62-433a-b69d-b2122d699c18","Type":"ContainerDied","Data":"345428ff1a7de884aaae1827955f4d6ead964252b308b325c27e1beb8463cc52"}
Sep 10 21:02:50.030726 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:50.030690    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w" event={"ID":"1863177c-3d62-433a-b69d-b2122d699c18","Type":"ContainerStarted","Data":"6c552c0841bf68658e3b3f117536df8dba622677d0a493a173c6f7af100a7fc0"}
Sep 10 21:02:50.031115 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:50.030813    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:02:50.061210 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:02:50.061161    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w" podStartSLOduration=1.792021487 podStartE2EDuration="6.061143871s" totalImagesPullingTime="3.220498382s" totalInitContainerRuntime="1.048624002s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:50.059051159 +0000 UTC m=+311.034711218" watchObservedRunningTime="2026-09-10 21:02:50.061143871 +0000 UTC m=+311.036803920"
Sep 10 21:03:01.037966 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:01.037930    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-4zq9w"
Sep 10 21:03:03.540911 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.540874    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"]
Sep 10 21:03:03.544623 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.544602    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"
Sep 10 21:03:03.548145 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.548119    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-47vpz\""
Sep 10 21:03:03.548289 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.548127    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 21:03:03.548732 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.548716    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 21:03:03.548795 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.548717    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 21:03:03.567109 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.567088    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"]
Sep 10 21:03:03.667436 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.667403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9a3479bb-9de5-4712-8c98-8f433b5a9317-manager-config\") pod \"lws-controller-manager-5d99b4bcb4-7cnkj\" (UID: \"9a3479bb-9de5-4712-8c98-8f433b5a9317\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"
Sep 10 21:03:03.667436 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.667443    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a3479bb-9de5-4712-8c98-8f433b5a9317-cert\") pod \"lws-controller-manager-5d99b4bcb4-7cnkj\" (UID: \"9a3479bb-9de5-4712-8c98-8f433b5a9317\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"
Sep 10 21:03:03.667649 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.667471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87sbg\" (UniqueName: \"kubernetes.io/projected/9a3479bb-9de5-4712-8c98-8f433b5a9317-kube-api-access-87sbg\") pod \"lws-controller-manager-5d99b4bcb4-7cnkj\" (UID: \"9a3479bb-9de5-4712-8c98-8f433b5a9317\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"
Sep 10 21:03:03.667649 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.667539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a3479bb-9de5-4712-8c98-8f433b5a9317-metrics-cert\") pod \"lws-controller-manager-5d99b4bcb4-7cnkj\" (UID: \"9a3479bb-9de5-4712-8c98-8f433b5a9317\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"
Sep 10 21:03:03.768239 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.768183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a3479bb-9de5-4712-8c98-8f433b5a9317-metrics-cert\") pod \"lws-controller-manager-5d99b4bcb4-7cnkj\" (UID: \"9a3479bb-9de5-4712-8c98-8f433b5a9317\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"
Sep 10 21:03:03.768405 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.768277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9a3479bb-9de5-4712-8c98-8f433b5a9317-manager-config\") pod \"lws-controller-manager-5d99b4bcb4-7cnkj\" (UID: \"9a3479bb-9de5-4712-8c98-8f433b5a9317\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"
Sep 10 21:03:03.768405 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.768310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a3479bb-9de5-4712-8c98-8f433b5a9317-cert\") pod \"lws-controller-manager-5d99b4bcb4-7cnkj\" (UID: \"9a3479bb-9de5-4712-8c98-8f433b5a9317\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"
Sep 10 21:03:03.768405 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.768348    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87sbg\" (UniqueName: \"kubernetes.io/projected/9a3479bb-9de5-4712-8c98-8f433b5a9317-kube-api-access-87sbg\") pod \"lws-controller-manager-5d99b4bcb4-7cnkj\" (UID: \"9a3479bb-9de5-4712-8c98-8f433b5a9317\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"
Sep 10 21:03:03.768957 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.768938    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9a3479bb-9de5-4712-8c98-8f433b5a9317-manager-config\") pod \"lws-controller-manager-5d99b4bcb4-7cnkj\" (UID: \"9a3479bb-9de5-4712-8c98-8f433b5a9317\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"
Sep 10 21:03:03.770655 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.770633    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a3479bb-9de5-4712-8c98-8f433b5a9317-cert\") pod \"lws-controller-manager-5d99b4bcb4-7cnkj\" (UID: \"9a3479bb-9de5-4712-8c98-8f433b5a9317\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"
Sep 10 21:03:03.770748 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.770728    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a3479bb-9de5-4712-8c98-8f433b5a9317-metrics-cert\") pod \"lws-controller-manager-5d99b4bcb4-7cnkj\" (UID: \"9a3479bb-9de5-4712-8c98-8f433b5a9317\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"
Sep 10 21:03:03.780362 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.780333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87sbg\" (UniqueName: \"kubernetes.io/projected/9a3479bb-9de5-4712-8c98-8f433b5a9317-kube-api-access-87sbg\") pod \"lws-controller-manager-5d99b4bcb4-7cnkj\" (UID: \"9a3479bb-9de5-4712-8c98-8f433b5a9317\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"
Sep 10 21:03:03.854721 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:03.854647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"
Sep 10 21:03:04.166588 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.166500    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-f4cb8c76f-6zxjc"]
Sep 10 21:03:04.170572 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.170550    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f4cb8c76f-6zxjc"
Sep 10 21:03:04.178330 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.178305    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 21:03:04.178632 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.178618    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 21:03:04.178819 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.178800    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 21:03:04.178984 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.178970    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 21:03:04.180650 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.180627    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"]
Sep 10 21:03:04.184380 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:03:04.184356    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3479bb_9de5_4712_8c98_8f433b5a9317.slice/crio-0cc73630b1a712920cad54ada21266e968200ebaa1e392d563ee064b6e67b89e WatchSource:0}: Error finding container 0cc73630b1a712920cad54ada21266e968200ebaa1e392d563ee064b6e67b89e: Status 404 returned error can't find the container with id 0cc73630b1a712920cad54ada21266e968200ebaa1e392d563ee064b6e67b89e
Sep 10 21:03:04.198108 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.198067    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-57zhj\""
Sep 10 21:03:04.200991 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.200970    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f4cb8c76f-6zxjc"]
Sep 10 21:03:04.271530 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.271500    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d102d1d2-c7d1-4886-a79a-607e03ef3122-tls-certs\") pod \"kube-auth-proxy-f4cb8c76f-6zxjc\" (UID: \"d102d1d2-c7d1-4886-a79a-607e03ef3122\") " pod="openshift-ingress/kube-auth-proxy-f4cb8c76f-6zxjc"
Sep 10 21:03:04.271684 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.271561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knvlt\" (UniqueName: \"kubernetes.io/projected/d102d1d2-c7d1-4886-a79a-607e03ef3122-kube-api-access-knvlt\") pod \"kube-auth-proxy-f4cb8c76f-6zxjc\" (UID: \"d102d1d2-c7d1-4886-a79a-607e03ef3122\") " pod="openshift-ingress/kube-auth-proxy-f4cb8c76f-6zxjc"
Sep 10 21:03:04.271684 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.271627    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d102d1d2-c7d1-4886-a79a-607e03ef3122-tmp\") pod \"kube-auth-proxy-f4cb8c76f-6zxjc\" (UID: \"d102d1d2-c7d1-4886-a79a-607e03ef3122\") " pod="openshift-ingress/kube-auth-proxy-f4cb8c76f-6zxjc"
Sep 10 21:03:04.372037 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.372002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d102d1d2-c7d1-4886-a79a-607e03ef3122-tmp\") pod \"kube-auth-proxy-f4cb8c76f-6zxjc\" (UID: \"d102d1d2-c7d1-4886-a79a-607e03ef3122\") " pod="openshift-ingress/kube-auth-proxy-f4cb8c76f-6zxjc"
Sep 10 21:03:04.372037 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.372038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d102d1d2-c7d1-4886-a79a-607e03ef3122-tls-certs\") pod \"kube-auth-proxy-f4cb8c76f-6zxjc\" (UID: \"d102d1d2-c7d1-4886-a79a-607e03ef3122\") " pod="openshift-ingress/kube-auth-proxy-f4cb8c76f-6zxjc"
Sep 10 21:03:04.372241 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.372085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knvlt\" (UniqueName: \"kubernetes.io/projected/d102d1d2-c7d1-4886-a79a-607e03ef3122-kube-api-access-knvlt\") pod \"kube-auth-proxy-f4cb8c76f-6zxjc\" (UID: \"d102d1d2-c7d1-4886-a79a-607e03ef3122\") " pod="openshift-ingress/kube-auth-proxy-f4cb8c76f-6zxjc"
Sep 10 21:03:04.374316 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.374292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d102d1d2-c7d1-4886-a79a-607e03ef3122-tmp\") pod \"kube-auth-proxy-f4cb8c76f-6zxjc\" (UID: \"d102d1d2-c7d1-4886-a79a-607e03ef3122\") " pod="openshift-ingress/kube-auth-proxy-f4cb8c76f-6zxjc"
Sep 10 21:03:04.374441 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.374422    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d102d1d2-c7d1-4886-a79a-607e03ef3122-tls-certs\") pod \"kube-auth-proxy-f4cb8c76f-6zxjc\" (UID: \"d102d1d2-c7d1-4886-a79a-607e03ef3122\") " pod="openshift-ingress/kube-auth-proxy-f4cb8c76f-6zxjc"
Sep 10 21:03:04.382083 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.382062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knvlt\" (UniqueName: \"kubernetes.io/projected/d102d1d2-c7d1-4886-a79a-607e03ef3122-kube-api-access-knvlt\") pod \"kube-auth-proxy-f4cb8c76f-6zxjc\" (UID: \"d102d1d2-c7d1-4886-a79a-607e03ef3122\") " pod="openshift-ingress/kube-auth-proxy-f4cb8c76f-6zxjc"
Sep 10 21:03:04.482418 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.482347    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f4cb8c76f-6zxjc"
Sep 10 21:03:04.754481 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:04.754454    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f4cb8c76f-6zxjc"]
Sep 10 21:03:04.756364 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:03:04.756341    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd102d1d2_c7d1_4886_a79a_607e03ef3122.slice/crio-696f7afc92b56e442071e1ee2b53efa3366201df198dfbe5e2ff68d7f3e88e37 WatchSource:0}: Error finding container 696f7afc92b56e442071e1ee2b53efa3366201df198dfbe5e2ff68d7f3e88e37: Status 404 returned error can't find the container with id 696f7afc92b56e442071e1ee2b53efa3366201df198dfbe5e2ff68d7f3e88e37
Sep 10 21:03:05.096524 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:05.096491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f4cb8c76f-6zxjc" event={"ID":"d102d1d2-c7d1-4886-a79a-607e03ef3122","Type":"ContainerStarted","Data":"696f7afc92b56e442071e1ee2b53efa3366201df198dfbe5e2ff68d7f3e88e37"}
Sep 10 21:03:05.097625 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:05.097601    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj" event={"ID":"9a3479bb-9de5-4712-8c98-8f433b5a9317","Type":"ContainerStarted","Data":"0cc73630b1a712920cad54ada21266e968200ebaa1e392d563ee064b6e67b89e"}
Sep 10 21:03:08.117097 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:08.117059    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f4cb8c76f-6zxjc" event={"ID":"d102d1d2-c7d1-4886-a79a-607e03ef3122","Type":"ContainerStarted","Data":"0c39175a668cd988ca84c9fd8fc6cbb7ff0670ecc51dd1884c73d5e34d824b3a"}
Sep 10 21:03:08.118782 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:08.118752    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj" event={"ID":"9a3479bb-9de5-4712-8c98-8f433b5a9317","Type":"ContainerStarted","Data":"9301f4d25dc7e5364dcec4c2f93aec3fa06429af72e8c94f89383230fd281036"}
Sep 10 21:03:08.118915 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:08.118867    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"
Sep 10 21:03:08.135919 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:08.135879    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-f4cb8c76f-6zxjc" podStartSLOduration=1.14560215 podStartE2EDuration="4.135865999s" totalImagesPullingTime="2.990263849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:08.13552611 +0000 UTC m=+329.111186158" watchObservedRunningTime="2026-09-10 21:03:08.135865999 +0000 UTC m=+329.111526046"
Sep 10 21:03:08.159712 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:08.159668    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj" podStartSLOduration=1.658765925 podStartE2EDuration="5.15965309s" totalImagesPullingTime="3.500887165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:08.159093951 +0000 UTC m=+329.134753999" watchObservedRunningTime="2026-09-10 21:03:08.15965309 +0000 UTC m=+329.135313136"
Sep 10 21:03:19.125181 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:03:19.125144    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-7cnkj"
Sep 10 21:04:30.486133 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:30.486091    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58b4874f44-5sm6k"]
Sep 10 21:04:44.177408 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.177373    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-drx8l"]
Sep 10 21:04:44.180868 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.180850    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drx8l"
Sep 10 21:04:44.182930 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.182906    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 21:04:44.183026 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.182999    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 21:04:44.183588 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.183565    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 21:04:44.183695 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.183607    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 21:04:44.183695 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.183670    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4zp26\""
Sep 10 21:04:44.190613 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.190593    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-drx8l"]
Sep 10 21:04:44.307265 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.307226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zvmg\" (UniqueName: \"kubernetes.io/projected/aea7f7ed-8fb3-447c-b5dd-5e5da268a467-kube-api-access-9zvmg\") pod \"kuadrant-console-plugin-fbdd4f445-drx8l\" (UID: \"aea7f7ed-8fb3-447c-b5dd-5e5da268a467\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drx8l"
Sep 10 21:04:44.307448 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.307309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aea7f7ed-8fb3-447c-b5dd-5e5da268a467-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-drx8l\" (UID: \"aea7f7ed-8fb3-447c-b5dd-5e5da268a467\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drx8l"
Sep 10 21:04:44.307448 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.307370    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/aea7f7ed-8fb3-447c-b5dd-5e5da268a467-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-drx8l\" (UID: \"aea7f7ed-8fb3-447c-b5dd-5e5da268a467\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drx8l"
Sep 10 21:04:44.408274 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.408233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aea7f7ed-8fb3-447c-b5dd-5e5da268a467-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-drx8l\" (UID: \"aea7f7ed-8fb3-447c-b5dd-5e5da268a467\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drx8l"
Sep 10 21:04:44.408429 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.408287    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/aea7f7ed-8fb3-447c-b5dd-5e5da268a467-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-drx8l\" (UID: \"aea7f7ed-8fb3-447c-b5dd-5e5da268a467\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drx8l"
Sep 10 21:04:44.408429 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.408339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zvmg\" (UniqueName: \"kubernetes.io/projected/aea7f7ed-8fb3-447c-b5dd-5e5da268a467-kube-api-access-9zvmg\") pod \"kuadrant-console-plugin-fbdd4f445-drx8l\" (UID: \"aea7f7ed-8fb3-447c-b5dd-5e5da268a467\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drx8l"
Sep 10 21:04:44.409006 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.408985    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aea7f7ed-8fb3-447c-b5dd-5e5da268a467-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-drx8l\" (UID: \"aea7f7ed-8fb3-447c-b5dd-5e5da268a467\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drx8l"
Sep 10 21:04:44.410692 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.410674    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/aea7f7ed-8fb3-447c-b5dd-5e5da268a467-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-drx8l\" (UID: \"aea7f7ed-8fb3-447c-b5dd-5e5da268a467\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drx8l"
Sep 10 21:04:44.416288 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.416269    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zvmg\" (UniqueName: \"kubernetes.io/projected/aea7f7ed-8fb3-447c-b5dd-5e5da268a467-kube-api-access-9zvmg\") pod \"kuadrant-console-plugin-fbdd4f445-drx8l\" (UID: \"aea7f7ed-8fb3-447c-b5dd-5e5da268a467\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drx8l"
Sep 10 21:04:44.491037 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.490964    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drx8l"
Sep 10 21:04:44.762573 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:44.762499    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-drx8l"]
Sep 10 21:04:44.766162 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:04:44.766133    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaea7f7ed_8fb3_447c_b5dd_5e5da268a467.slice/crio-b2ea35623821b647d54d77e6e22a53b7f71bf42c34a06dba3a17646acaafb521 WatchSource:0}: Error finding container b2ea35623821b647d54d77e6e22a53b7f71bf42c34a06dba3a17646acaafb521: Status 404 returned error can't find the container with id b2ea35623821b647d54d77e6e22a53b7f71bf42c34a06dba3a17646acaafb521
Sep 10 21:04:45.545381 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:45.545318    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drx8l" event={"ID":"aea7f7ed-8fb3-447c-b5dd-5e5da268a467","Type":"ContainerStarted","Data":"b2ea35623821b647d54d77e6e22a53b7f71bf42c34a06dba3a17646acaafb521"}
Sep 10 21:04:49.573719 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:49.573685    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drx8l" event={"ID":"aea7f7ed-8fb3-447c-b5dd-5e5da268a467","Type":"ContainerStarted","Data":"6706a4c88d63dde6bb01d7b7f09c77da2566db0bf0479c37a6485809f209875e"}
Sep 10 21:04:49.590831 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:49.590781    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drx8l" podStartSLOduration=1.341332782 podStartE2EDuration="5.590765966s" totalImagesPullingTime="4.249433184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:49.589090637 +0000 UTC m=+430.564750685" watchObservedRunningTime="2026-09-10 21:04:49.590765966 +0000 UTC m=+430.566426012"
Sep 10 21:04:55.505399 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:55.505360    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58b4874f44-5sm6k" podUID="ede14eb3-2e94-4ae0-8d58-4f4b627e3f43" containerName="console" containerID="cri-o://e865e771734ad3ae4a1e60e5f917fdd38248a0f9c582757c588a041f3a389e20" gracePeriod=15
Sep 10 21:04:55.747310 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:55.747288    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:04:55.914706 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:55.914662    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-serving-cert\") pod \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") "
Sep 10 21:04:55.914887 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:55.914719    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-service-ca\") pod \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") "
Sep 10 21:04:55.914887 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:55.914746    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-config\") pod \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") "
Sep 10 21:04:55.914887 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:55.914776    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-oauth-serving-cert\") pod \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") "
Sep 10 21:04:55.914887 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:55.914832    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzk7f\" (UniqueName: \"kubernetes.io/projected/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-kube-api-access-zzk7f\") pod \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") "
Sep 10 21:04:55.914887 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:55.914864    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-oauth-config\") pod \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") "
Sep 10 21:04:55.915145 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:55.914905    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-trusted-ca-bundle\") pod \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\" (UID: \"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43\") "
Sep 10 21:04:55.915283 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:55.915254    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-service-ca" (OuterVolumeSpecName: "service-ca") pod "ede14eb3-2e94-4ae0-8d58-4f4b627e3f43" (UID: "ede14eb3-2e94-4ae0-8d58-4f4b627e3f43"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:55.915283 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:55.915270    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ede14eb3-2e94-4ae0-8d58-4f4b627e3f43" (UID: "ede14eb3-2e94-4ae0-8d58-4f4b627e3f43"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:55.915407 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:55.915265    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-config" (OuterVolumeSpecName: "console-config") pod "ede14eb3-2e94-4ae0-8d58-4f4b627e3f43" (UID: "ede14eb3-2e94-4ae0-8d58-4f4b627e3f43"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:55.915828 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:55.915803    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ede14eb3-2e94-4ae0-8d58-4f4b627e3f43" (UID: "ede14eb3-2e94-4ae0-8d58-4f4b627e3f43"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:55.917042 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:55.917014    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ede14eb3-2e94-4ae0-8d58-4f4b627e3f43" (UID: "ede14eb3-2e94-4ae0-8d58-4f4b627e3f43"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:04:55.917122 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:55.917026    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ede14eb3-2e94-4ae0-8d58-4f4b627e3f43" (UID: "ede14eb3-2e94-4ae0-8d58-4f4b627e3f43"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:04:55.917122 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:55.917048    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-kube-api-access-zzk7f" (OuterVolumeSpecName: "kube-api-access-zzk7f") pod "ede14eb3-2e94-4ae0-8d58-4f4b627e3f43" (UID: "ede14eb3-2e94-4ae0-8d58-4f4b627e3f43"). InnerVolumeSpecName "kube-api-access-zzk7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:04:56.015623 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:56.015573    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-serving-cert\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:04:56.015623 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:56.015618    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-service-ca\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:04:56.015623 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:56.015629    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-config\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:04:56.015623 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:56.015638    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-oauth-serving-cert\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:04:56.015894 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:56.015647    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzk7f\" (UniqueName: \"kubernetes.io/projected/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-kube-api-access-zzk7f\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:04:56.015894 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:56.015655    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-console-oauth-config\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:04:56.015894 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:56.015663    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43-trusted-ca-bundle\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:04:56.599739 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:56.599709    2581 generic.go:356] "Generic (PLEG): container finished" podID="ede14eb3-2e94-4ae0-8d58-4f4b627e3f43" containerID="e865e771734ad3ae4a1e60e5f917fdd38248a0f9c582757c588a041f3a389e20" exitCode=0
Sep 10 21:04:56.600100 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:56.599755    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b4874f44-5sm6k"
Sep 10 21:04:56.600100 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:56.599791    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b4874f44-5sm6k" event={"ID":"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43","Type":"ContainerDied","Data":"e865e771734ad3ae4a1e60e5f917fdd38248a0f9c582757c588a041f3a389e20"}
Sep 10 21:04:56.600100 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:56.599843    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b4874f44-5sm6k" event={"ID":"ede14eb3-2e94-4ae0-8d58-4f4b627e3f43","Type":"ContainerDied","Data":"c8d80db09fa53801760069c400c24d8e0b73788244807ddf73e37c6db58173f8"}
Sep 10 21:04:56.600100 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:56.599868    2581 scope.go:117] "RemoveContainer" containerID="e865e771734ad3ae4a1e60e5f917fdd38248a0f9c582757c588a041f3a389e20"
Sep 10 21:04:56.610965 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:56.610939    2581 scope.go:117] "RemoveContainer" containerID="e865e771734ad3ae4a1e60e5f917fdd38248a0f9c582757c588a041f3a389e20"
Sep 10 21:04:56.611301 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:04:56.611271    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e865e771734ad3ae4a1e60e5f917fdd38248a0f9c582757c588a041f3a389e20\": container with ID starting with e865e771734ad3ae4a1e60e5f917fdd38248a0f9c582757c588a041f3a389e20 not found: ID does not exist" containerID="e865e771734ad3ae4a1e60e5f917fdd38248a0f9c582757c588a041f3a389e20"
Sep 10 21:04:56.611387 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:56.611306    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e865e771734ad3ae4a1e60e5f917fdd38248a0f9c582757c588a041f3a389e20"} err="failed to get container status \"e865e771734ad3ae4a1e60e5f917fdd38248a0f9c582757c588a041f3a389e20\": rpc error: code = NotFound desc = could not find container \"e865e771734ad3ae4a1e60e5f917fdd38248a0f9c582757c588a041f3a389e20\": container with ID starting with e865e771734ad3ae4a1e60e5f917fdd38248a0f9c582757c588a041f3a389e20 not found: ID does not exist"
Sep 10 21:04:56.624723 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:56.624702    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58b4874f44-5sm6k"]
Sep 10 21:04:56.628189 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:56.628169    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58b4874f44-5sm6k"]
Sep 10 21:04:57.573906 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:04:57.573871    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ede14eb3-2e94-4ae0-8d58-4f4b627e3f43" path="/var/lib/kubelet/pods/ede14eb3-2e94-4ae0-8d58-4f4b627e3f43/volumes"
Sep 10 21:05:27.954663 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:27.954630    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lp7bh"]
Sep 10 21:05:27.955113 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:27.954977    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ede14eb3-2e94-4ae0-8d58-4f4b627e3f43" containerName="console"
Sep 10 21:05:27.955113 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:27.954988    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ede14eb3-2e94-4ae0-8d58-4f4b627e3f43" containerName="console"
Sep 10 21:05:27.955113 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:27.955064    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ede14eb3-2e94-4ae0-8d58-4f4b627e3f43" containerName="console"
Sep 10 21:05:27.975633 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:27.975606    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lp7bh"]
Sep 10 21:05:27.975752 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:27.975722    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lp7bh"
Sep 10 21:05:27.977968 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:27.977951    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 21:05:28.031301 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:28.031263    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lp7bh"]
Sep 10 21:05:28.080399 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:28.080369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f696841-01fa-4e63-8d1f-85891f01b8d2-config-file\") pod \"limitador-limitador-8588749977-lp7bh\" (UID: \"8f696841-01fa-4e63-8d1f-85891f01b8d2\") " pod="kuadrant-system/limitador-limitador-8588749977-lp7bh"
Sep 10 21:05:28.080543 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:28.080424    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx9hh\" (UniqueName: \"kubernetes.io/projected/8f696841-01fa-4e63-8d1f-85891f01b8d2-kube-api-access-hx9hh\") pod \"limitador-limitador-8588749977-lp7bh\" (UID: \"8f696841-01fa-4e63-8d1f-85891f01b8d2\") " pod="kuadrant-system/limitador-limitador-8588749977-lp7bh"
Sep 10 21:05:28.181519 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:28.181489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f696841-01fa-4e63-8d1f-85891f01b8d2-config-file\") pod \"limitador-limitador-8588749977-lp7bh\" (UID: \"8f696841-01fa-4e63-8d1f-85891f01b8d2\") " pod="kuadrant-system/limitador-limitador-8588749977-lp7bh"
Sep 10 21:05:28.181666 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:28.181542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx9hh\" (UniqueName: \"kubernetes.io/projected/8f696841-01fa-4e63-8d1f-85891f01b8d2-kube-api-access-hx9hh\") pod \"limitador-limitador-8588749977-lp7bh\" (UID: \"8f696841-01fa-4e63-8d1f-85891f01b8d2\") " pod="kuadrant-system/limitador-limitador-8588749977-lp7bh"
Sep 10 21:05:28.182086 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:28.182059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f696841-01fa-4e63-8d1f-85891f01b8d2-config-file\") pod \"limitador-limitador-8588749977-lp7bh\" (UID: \"8f696841-01fa-4e63-8d1f-85891f01b8d2\") " pod="kuadrant-system/limitador-limitador-8588749977-lp7bh"
Sep 10 21:05:28.190277 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:28.190251    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx9hh\" (UniqueName: \"kubernetes.io/projected/8f696841-01fa-4e63-8d1f-85891f01b8d2-kube-api-access-hx9hh\") pod \"limitador-limitador-8588749977-lp7bh\" (UID: \"8f696841-01fa-4e63-8d1f-85891f01b8d2\") " pod="kuadrant-system/limitador-limitador-8588749977-lp7bh"
Sep 10 21:05:28.288065 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:28.288036    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lp7bh"
Sep 10 21:05:28.569968 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:28.569893    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lp7bh"]
Sep 10 21:05:28.575981 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:05:28.575945    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f696841_01fa_4e63_8d1f_85891f01b8d2.slice/crio-3b2a9b6733b4e08776be2456f54dba6b7a7888073553400016e421aa95442dd0 WatchSource:0}: Error finding container 3b2a9b6733b4e08776be2456f54dba6b7a7888073553400016e421aa95442dd0: Status 404 returned error can't find the container with id 3b2a9b6733b4e08776be2456f54dba6b7a7888073553400016e421aa95442dd0
Sep 10 21:05:28.738734 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:28.738699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lp7bh" event={"ID":"8f696841-01fa-4e63-8d1f-85891f01b8d2","Type":"ContainerStarted","Data":"3b2a9b6733b4e08776be2456f54dba6b7a7888073553400016e421aa95442dd0"}
Sep 10 21:05:29.000136 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:29.000059    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-b6bcl"]
Sep 10 21:05:29.004788 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:29.004770    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-b6bcl"
Sep 10 21:05:29.007158 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:29.007137    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t4rfj\""
Sep 10 21:05:29.013594 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:29.013574    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-b6bcl"]
Sep 10 21:05:29.089573 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:29.089541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdtpg\" (UniqueName: \"kubernetes.io/projected/91908f07-e860-4fd8-bed1-ec46f8df0862-kube-api-access-fdtpg\") pod \"authorino-59c974bd48-b6bcl\" (UID: \"91908f07-e860-4fd8-bed1-ec46f8df0862\") " pod="kuadrant-system/authorino-59c974bd48-b6bcl"
Sep 10 21:05:29.190477 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:29.190442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdtpg\" (UniqueName: \"kubernetes.io/projected/91908f07-e860-4fd8-bed1-ec46f8df0862-kube-api-access-fdtpg\") pod \"authorino-59c974bd48-b6bcl\" (UID: \"91908f07-e860-4fd8-bed1-ec46f8df0862\") " pod="kuadrant-system/authorino-59c974bd48-b6bcl"
Sep 10 21:05:29.202588 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:29.202550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdtpg\" (UniqueName: \"kubernetes.io/projected/91908f07-e860-4fd8-bed1-ec46f8df0862-kube-api-access-fdtpg\") pod \"authorino-59c974bd48-b6bcl\" (UID: \"91908f07-e860-4fd8-bed1-ec46f8df0862\") " pod="kuadrant-system/authorino-59c974bd48-b6bcl"
Sep 10 21:05:29.314430 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:29.314393    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-b6bcl"
Sep 10 21:05:29.597096 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:29.597069    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-b6bcl"]
Sep 10 21:05:29.599996 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:05:29.599967    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91908f07_e860_4fd8_bed1_ec46f8df0862.slice/crio-1052f25660869585783c766edc7df03aaa755381af91e434c7a6f7ef7b11f0bc WatchSource:0}: Error finding container 1052f25660869585783c766edc7df03aaa755381af91e434c7a6f7ef7b11f0bc: Status 404 returned error can't find the container with id 1052f25660869585783c766edc7df03aaa755381af91e434c7a6f7ef7b11f0bc
Sep 10 21:05:29.745838 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:29.745803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-b6bcl" event={"ID":"91908f07-e860-4fd8-bed1-ec46f8df0862","Type":"ContainerStarted","Data":"1052f25660869585783c766edc7df03aaa755381af91e434c7a6f7ef7b11f0bc"}
Sep 10 21:05:30.752306 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:30.752265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lp7bh" event={"ID":"8f696841-01fa-4e63-8d1f-85891f01b8d2","Type":"ContainerStarted","Data":"eb0a5d033c4f3c49b8624226d909ecc84f6c3b6f24a0e1eac69c349d0eb48fcb"}
Sep 10 21:05:30.752744 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:30.752369    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-lp7bh"
Sep 10 21:05:30.776640 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:30.776594    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-lp7bh" podStartSLOduration=1.809120077 podStartE2EDuration="3.776575185s" totalImagesPullingTime="1.967455108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:05:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:05:30.776031593 +0000 UTC m=+471.751691639" watchObservedRunningTime="2026-09-10 21:05:30.776575185 +0000 UTC m=+471.752235235"
Sep 10 21:05:31.758695 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:31.758609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-b6bcl" event={"ID":"91908f07-e860-4fd8-bed1-ec46f8df0862","Type":"ContainerStarted","Data":"c8361af4709c8a88a7b2d1ea852ef5da7f19948cbb60c6ca4e4a13469949d5fd"}
Sep 10 21:05:31.779579 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:31.779529    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-b6bcl" podStartSLOduration=1.8728620710000001 podStartE2EDuration="3.77951572s" totalImagesPullingTime="1.906653649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:05:31.77789108 +0000 UTC m=+472.753551149" watchObservedRunningTime="2026-09-10 21:05:31.77951572 +0000 UTC m=+472.755175767"
Sep 10 21:05:41.760064 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:41.760034    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-lp7bh"
Sep 10 21:05:41.863285 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:41.863254    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lp7bh"]
Sep 10 21:05:41.863496 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:41.863448    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-lp7bh" podUID="8f696841-01fa-4e63-8d1f-85891f01b8d2" containerName="limitador" containerID="cri-o://eb0a5d033c4f3c49b8624226d909ecc84f6c3b6f24a0e1eac69c349d0eb48fcb" gracePeriod=30
Sep 10 21:05:42.797667 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:42.797642    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lp7bh"
Sep 10 21:05:42.807817 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:42.807791    2581 generic.go:356] "Generic (PLEG): container finished" podID="8f696841-01fa-4e63-8d1f-85891f01b8d2" containerID="eb0a5d033c4f3c49b8624226d909ecc84f6c3b6f24a0e1eac69c349d0eb48fcb" exitCode=0
Sep 10 21:05:42.807933 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:42.807829    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lp7bh"
Sep 10 21:05:42.807933 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:42.807833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lp7bh" event={"ID":"8f696841-01fa-4e63-8d1f-85891f01b8d2","Type":"ContainerDied","Data":"eb0a5d033c4f3c49b8624226d909ecc84f6c3b6f24a0e1eac69c349d0eb48fcb"}
Sep 10 21:05:42.807933 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:42.807885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lp7bh" event={"ID":"8f696841-01fa-4e63-8d1f-85891f01b8d2","Type":"ContainerDied","Data":"3b2a9b6733b4e08776be2456f54dba6b7a7888073553400016e421aa95442dd0"}
Sep 10 21:05:42.807933 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:42.807908    2581 scope.go:117] "RemoveContainer" containerID="eb0a5d033c4f3c49b8624226d909ecc84f6c3b6f24a0e1eac69c349d0eb48fcb"
Sep 10 21:05:42.815709 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:42.815694    2581 scope.go:117] "RemoveContainer" containerID="eb0a5d033c4f3c49b8624226d909ecc84f6c3b6f24a0e1eac69c349d0eb48fcb"
Sep 10 21:05:42.815977 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:05:42.815958    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb0a5d033c4f3c49b8624226d909ecc84f6c3b6f24a0e1eac69c349d0eb48fcb\": container with ID starting with eb0a5d033c4f3c49b8624226d909ecc84f6c3b6f24a0e1eac69c349d0eb48fcb not found: ID does not exist" containerID="eb0a5d033c4f3c49b8624226d909ecc84f6c3b6f24a0e1eac69c349d0eb48fcb"
Sep 10 21:05:42.816034 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:42.815984    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb0a5d033c4f3c49b8624226d909ecc84f6c3b6f24a0e1eac69c349d0eb48fcb"} err="failed to get container status \"eb0a5d033c4f3c49b8624226d909ecc84f6c3b6f24a0e1eac69c349d0eb48fcb\": rpc error: code = NotFound desc = could not find container \"eb0a5d033c4f3c49b8624226d909ecc84f6c3b6f24a0e1eac69c349d0eb48fcb\": container with ID starting with eb0a5d033c4f3c49b8624226d909ecc84f6c3b6f24a0e1eac69c349d0eb48fcb not found: ID does not exist"
Sep 10 21:05:42.911741 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:42.911708    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f696841-01fa-4e63-8d1f-85891f01b8d2-config-file\") pod \"8f696841-01fa-4e63-8d1f-85891f01b8d2\" (UID: \"8f696841-01fa-4e63-8d1f-85891f01b8d2\") "
Sep 10 21:05:42.911907 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:42.911832    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx9hh\" (UniqueName: \"kubernetes.io/projected/8f696841-01fa-4e63-8d1f-85891f01b8d2-kube-api-access-hx9hh\") pod \"8f696841-01fa-4e63-8d1f-85891f01b8d2\" (UID: \"8f696841-01fa-4e63-8d1f-85891f01b8d2\") "
Sep 10 21:05:42.912129 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:42.912108    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f696841-01fa-4e63-8d1f-85891f01b8d2-config-file" (OuterVolumeSpecName: "config-file") pod "8f696841-01fa-4e63-8d1f-85891f01b8d2" (UID: "8f696841-01fa-4e63-8d1f-85891f01b8d2"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:05:42.914027 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:42.914007    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f696841-01fa-4e63-8d1f-85891f01b8d2-kube-api-access-hx9hh" (OuterVolumeSpecName: "kube-api-access-hx9hh") pod "8f696841-01fa-4e63-8d1f-85891f01b8d2" (UID: "8f696841-01fa-4e63-8d1f-85891f01b8d2"). InnerVolumeSpecName "kube-api-access-hx9hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:43.013358 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:43.013283    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx9hh\" (UniqueName: \"kubernetes.io/projected/8f696841-01fa-4e63-8d1f-85891f01b8d2-kube-api-access-hx9hh\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:05:43.013358 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:43.013311    2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f696841-01fa-4e63-8d1f-85891f01b8d2-config-file\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:05:43.128652 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:43.128624    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lp7bh"]
Sep 10 21:05:43.133040 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:43.133019    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lp7bh"]
Sep 10 21:05:43.572993 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:43.572960    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f696841-01fa-4e63-8d1f-85891f01b8d2" path="/var/lib/kubelet/pods/8f696841-01fa-4e63-8d1f-85891f01b8d2/volumes"
Sep 10 21:05:44.582064 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:44.582031    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-g6c8r"]
Sep 10 21:05:44.582610 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:44.582594    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f696841-01fa-4e63-8d1f-85891f01b8d2" containerName="limitador"
Sep 10 21:05:44.582668 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:44.582614    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f696841-01fa-4e63-8d1f-85891f01b8d2" containerName="limitador"
Sep 10 21:05:44.582703 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:44.582693    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f696841-01fa-4e63-8d1f-85891f01b8d2" containerName="limitador"
Sep 10 21:05:44.587325 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:44.587296    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6c8r"
Sep 10 21:05:44.589678 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:44.589656    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 21:05:44.589801 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:44.589705    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:05:44.589933 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:44.589790    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-mrspk\""
Sep 10 21:05:44.590341 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:44.590325    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 21:05:44.595289 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:44.595268    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-g6c8r"]
Sep 10 21:05:44.728689 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:44.728661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5cb95453-59ad-45eb-aa0c-11262807be37-data\") pod \"postgres-69fb4594cf-g6c8r\" (UID: \"5cb95453-59ad-45eb-aa0c-11262807be37\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6c8r"
Sep 10 21:05:44.728689 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:44.728692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbzwc\" (UniqueName: \"kubernetes.io/projected/5cb95453-59ad-45eb-aa0c-11262807be37-kube-api-access-vbzwc\") pod \"postgres-69fb4594cf-g6c8r\" (UID: \"5cb95453-59ad-45eb-aa0c-11262807be37\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6c8r"
Sep 10 21:05:44.830018 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:44.829989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5cb95453-59ad-45eb-aa0c-11262807be37-data\") pod \"postgres-69fb4594cf-g6c8r\" (UID: \"5cb95453-59ad-45eb-aa0c-11262807be37\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6c8r"
Sep 10 21:05:44.830018 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:44.830019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbzwc\" (UniqueName: \"kubernetes.io/projected/5cb95453-59ad-45eb-aa0c-11262807be37-kube-api-access-vbzwc\") pod \"postgres-69fb4594cf-g6c8r\" (UID: \"5cb95453-59ad-45eb-aa0c-11262807be37\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6c8r"
Sep 10 21:05:44.830355 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:44.830338    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5cb95453-59ad-45eb-aa0c-11262807be37-data\") pod \"postgres-69fb4594cf-g6c8r\" (UID: \"5cb95453-59ad-45eb-aa0c-11262807be37\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6c8r"
Sep 10 21:05:44.842158 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:44.842102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbzwc\" (UniqueName: \"kubernetes.io/projected/5cb95453-59ad-45eb-aa0c-11262807be37-kube-api-access-vbzwc\") pod \"postgres-69fb4594cf-g6c8r\" (UID: \"5cb95453-59ad-45eb-aa0c-11262807be37\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6c8r"
Sep 10 21:05:44.898911 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:44.898883    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6c8r"
Sep 10 21:05:45.168478 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:45.168455    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-g6c8r"]
Sep 10 21:05:45.170832 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:05:45.170806    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb95453_59ad_45eb_aa0c_11262807be37.slice/crio-f4b256b6930262311e21035110f7101665a89a8d059394454b2b3186c3baae43 WatchSource:0}: Error finding container f4b256b6930262311e21035110f7101665a89a8d059394454b2b3186c3baae43: Status 404 returned error can't find the container with id f4b256b6930262311e21035110f7101665a89a8d059394454b2b3186c3baae43
Sep 10 21:05:45.822728 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:45.822693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6c8r" event={"ID":"5cb95453-59ad-45eb-aa0c-11262807be37","Type":"ContainerStarted","Data":"f4b256b6930262311e21035110f7101665a89a8d059394454b2b3186c3baae43"}
Sep 10 21:05:47.834648 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:47.834611    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6c8r" event={"ID":"5cb95453-59ad-45eb-aa0c-11262807be37","Type":"ContainerStarted","Data":"2207652e560a7a7cce1885c114c2effaa9071ca8d01577c4262e52d0f7c28c7b"}
Sep 10 21:05:47.835090 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:47.834693    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6c8r"
Sep 10 21:05:47.854170 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:47.854127    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6c8r" podStartSLOduration=1.766794056 podStartE2EDuration="3.854111379s" totalImagesPullingTime="2.087317323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:05:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:05:47.852858816 +0000 UTC m=+488.828518864" watchObservedRunningTime="2026-09-10 21:05:47.854111379 +0000 UTC m=+488.829771429"
Sep 10 21:05:53.868549 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:53.868514    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6c8r"
Sep 10 21:05:54.528583 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:54.528548    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ssfls"]
Sep 10 21:05:54.535414 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:54.535395    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-ssfls"
Sep 10 21:05:54.537551 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:54.537531    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 21:05:54.542547 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:54.542519    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ssfls"]
Sep 10 21:05:54.707240 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:54.707196    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ssfls"]
Sep 10 21:05:54.707479 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:05:54.707461    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2tlxn tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-ssfls" podUID="88be7f88-5b4a-4698-b98f-c0a86135c010"
Sep 10 21:05:54.719322 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:54.719291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tlxn\" (UniqueName: \"kubernetes.io/projected/88be7f88-5b4a-4698-b98f-c0a86135c010-kube-api-access-2tlxn\") pod \"authorino-6d4dfc649f-ssfls\" (UID: \"88be7f88-5b4a-4698-b98f-c0a86135c010\") " pod="kuadrant-system/authorino-6d4dfc649f-ssfls"
Sep 10 21:05:54.719438 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:54.719342    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88be7f88-5b4a-4698-b98f-c0a86135c010-tls-cert\") pod \"authorino-6d4dfc649f-ssfls\" (UID: \"88be7f88-5b4a-4698-b98f-c0a86135c010\") " pod="kuadrant-system/authorino-6d4dfc649f-ssfls"
Sep 10 21:05:54.819999 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:54.819974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tlxn\" (UniqueName: \"kubernetes.io/projected/88be7f88-5b4a-4698-b98f-c0a86135c010-kube-api-access-2tlxn\") pod \"authorino-6d4dfc649f-ssfls\" (UID: \"88be7f88-5b4a-4698-b98f-c0a86135c010\") " pod="kuadrant-system/authorino-6d4dfc649f-ssfls"
Sep 10 21:05:54.820139 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:54.820008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88be7f88-5b4a-4698-b98f-c0a86135c010-tls-cert\") pod \"authorino-6d4dfc649f-ssfls\" (UID: \"88be7f88-5b4a-4698-b98f-c0a86135c010\") " pod="kuadrant-system/authorino-6d4dfc649f-ssfls"
Sep 10 21:05:54.822304 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:54.822286    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88be7f88-5b4a-4698-b98f-c0a86135c010-tls-cert\") pod \"authorino-6d4dfc649f-ssfls\" (UID: \"88be7f88-5b4a-4698-b98f-c0a86135c010\") " pod="kuadrant-system/authorino-6d4dfc649f-ssfls"
Sep 10 21:05:54.829621 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:54.829602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tlxn\" (UniqueName: \"kubernetes.io/projected/88be7f88-5b4a-4698-b98f-c0a86135c010-kube-api-access-2tlxn\") pod \"authorino-6d4dfc649f-ssfls\" (UID: \"88be7f88-5b4a-4698-b98f-c0a86135c010\") " pod="kuadrant-system/authorino-6d4dfc649f-ssfls"
Sep 10 21:05:54.866216 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:54.866180    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-ssfls"
Sep 10 21:05:54.873924 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:54.873908    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-ssfls"
Sep 10 21:05:55.022274 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:55.022248    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tlxn\" (UniqueName: \"kubernetes.io/projected/88be7f88-5b4a-4698-b98f-c0a86135c010-kube-api-access-2tlxn\") pod \"88be7f88-5b4a-4698-b98f-c0a86135c010\" (UID: \"88be7f88-5b4a-4698-b98f-c0a86135c010\") "
Sep 10 21:05:55.022405 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:55.022283    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88be7f88-5b4a-4698-b98f-c0a86135c010-tls-cert\") pod \"88be7f88-5b4a-4698-b98f-c0a86135c010\" (UID: \"88be7f88-5b4a-4698-b98f-c0a86135c010\") "
Sep 10 21:05:55.024253 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:55.024222    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88be7f88-5b4a-4698-b98f-c0a86135c010-kube-api-access-2tlxn" (OuterVolumeSpecName: "kube-api-access-2tlxn") pod "88be7f88-5b4a-4698-b98f-c0a86135c010" (UID: "88be7f88-5b4a-4698-b98f-c0a86135c010"). InnerVolumeSpecName "kube-api-access-2tlxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:55.024353 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:55.024248    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88be7f88-5b4a-4698-b98f-c0a86135c010-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "88be7f88-5b4a-4698-b98f-c0a86135c010" (UID: "88be7f88-5b4a-4698-b98f-c0a86135c010"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:05:55.123619 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:55.123536    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tlxn\" (UniqueName: \"kubernetes.io/projected/88be7f88-5b4a-4698-b98f-c0a86135c010-kube-api-access-2tlxn\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:05:55.123619 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:55.123564    2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88be7f88-5b4a-4698-b98f-c0a86135c010-tls-cert\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:05:55.870279 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:55.870160    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-ssfls"
Sep 10 21:05:55.910586 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:55.910556    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ssfls"]
Sep 10 21:05:55.912345 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:55.912321    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ssfls"]
Sep 10 21:05:56.001771 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:56.001737    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-b6bcl"]
Sep 10 21:05:56.001930 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:56.001902    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-b6bcl" podUID="91908f07-e860-4fd8-bed1-ec46f8df0862" containerName="authorino" containerID="cri-o://c8361af4709c8a88a7b2d1ea852ef5da7f19948cbb60c6ca4e4a13469949d5fd" gracePeriod=30
Sep 10 21:05:56.246179 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:56.246156    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-b6bcl"
Sep 10 21:05:56.435478 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:56.435385    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdtpg\" (UniqueName: \"kubernetes.io/projected/91908f07-e860-4fd8-bed1-ec46f8df0862-kube-api-access-fdtpg\") pod \"91908f07-e860-4fd8-bed1-ec46f8df0862\" (UID: \"91908f07-e860-4fd8-bed1-ec46f8df0862\") "
Sep 10 21:05:56.437396 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:56.437360    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91908f07-e860-4fd8-bed1-ec46f8df0862-kube-api-access-fdtpg" (OuterVolumeSpecName: "kube-api-access-fdtpg") pod "91908f07-e860-4fd8-bed1-ec46f8df0862" (UID: "91908f07-e860-4fd8-bed1-ec46f8df0862"). InnerVolumeSpecName "kube-api-access-fdtpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:56.536177 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:56.536151    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdtpg\" (UniqueName: \"kubernetes.io/projected/91908f07-e860-4fd8-bed1-ec46f8df0862-kube-api-access-fdtpg\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:05:56.876287 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:56.876259    2581 generic.go:356] "Generic (PLEG): container finished" podID="91908f07-e860-4fd8-bed1-ec46f8df0862" containerID="c8361af4709c8a88a7b2d1ea852ef5da7f19948cbb60c6ca4e4a13469949d5fd" exitCode=0
Sep 10 21:05:56.876460 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:56.876301    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-b6bcl"
Sep 10 21:05:56.876460 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:56.876345    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-b6bcl" event={"ID":"91908f07-e860-4fd8-bed1-ec46f8df0862","Type":"ContainerDied","Data":"c8361af4709c8a88a7b2d1ea852ef5da7f19948cbb60c6ca4e4a13469949d5fd"}
Sep 10 21:05:56.876460 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:56.876381    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-b6bcl" event={"ID":"91908f07-e860-4fd8-bed1-ec46f8df0862","Type":"ContainerDied","Data":"1052f25660869585783c766edc7df03aaa755381af91e434c7a6f7ef7b11f0bc"}
Sep 10 21:05:56.876460 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:56.876397    2581 scope.go:117] "RemoveContainer" containerID="c8361af4709c8a88a7b2d1ea852ef5da7f19948cbb60c6ca4e4a13469949d5fd"
Sep 10 21:05:56.884539 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:56.884524    2581 scope.go:117] "RemoveContainer" containerID="c8361af4709c8a88a7b2d1ea852ef5da7f19948cbb60c6ca4e4a13469949d5fd"
Sep 10 21:05:56.884809 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:05:56.884788    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8361af4709c8a88a7b2d1ea852ef5da7f19948cbb60c6ca4e4a13469949d5fd\": container with ID starting with c8361af4709c8a88a7b2d1ea852ef5da7f19948cbb60c6ca4e4a13469949d5fd not found: ID does not exist" containerID="c8361af4709c8a88a7b2d1ea852ef5da7f19948cbb60c6ca4e4a13469949d5fd"
Sep 10 21:05:56.884856 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:56.884817    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8361af4709c8a88a7b2d1ea852ef5da7f19948cbb60c6ca4e4a13469949d5fd"} err="failed to get container status \"c8361af4709c8a88a7b2d1ea852ef5da7f19948cbb60c6ca4e4a13469949d5fd\": rpc error: code = NotFound desc = could not find container \"c8361af4709c8a88a7b2d1ea852ef5da7f19948cbb60c6ca4e4a13469949d5fd\": container with ID starting with c8361af4709c8a88a7b2d1ea852ef5da7f19948cbb60c6ca4e4a13469949d5fd not found: ID does not exist"
Sep 10 21:05:56.896786 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:56.896765    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-b6bcl"]
Sep 10 21:05:56.901609 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:56.901590    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-b6bcl"]
Sep 10 21:05:57.574282 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:57.574249    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88be7f88-5b4a-4698-b98f-c0a86135c010" path="/var/lib/kubelet/pods/88be7f88-5b4a-4698-b98f-c0a86135c010/volumes"
Sep 10 21:05:57.574646 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:05:57.574486    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91908f07-e860-4fd8-bed1-ec46f8df0862" path="/var/lib/kubelet/pods/91908f07-e860-4fd8-bed1-ec46f8df0862/volumes"
Sep 10 21:06:39.756356 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:39.756323    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"]
Sep 10 21:06:39.756739 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:39.756683    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91908f07-e860-4fd8-bed1-ec46f8df0862" containerName="authorino"
Sep 10 21:06:39.756739 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:39.756695    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="91908f07-e860-4fd8-bed1-ec46f8df0862" containerName="authorino"
Sep 10 21:06:39.756813 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:39.756760    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="91908f07-e860-4fd8-bed1-ec46f8df0862" containerName="authorino"
Sep 10 21:06:39.761179 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:39.761158    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:06:39.764176 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:39.764155    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rf2kh\""
Sep 10 21:06:39.764176 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:39.764169    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 21:06:39.764357 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:39.764162    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 21:06:39.771520 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:39.771500    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"]
Sep 10 21:06:39.900724 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:39.900687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nc52\" (UniqueName: \"kubernetes.io/projected/da254e82-d544-4632-b350-eb3fa670018c-kube-api-access-9nc52\") pod \"maas-api-5867c8d69-j8mzx\" (UID: \"da254e82-d544-4632-b350-eb3fa670018c\") " pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:06:39.900891 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:39.900769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da254e82-d544-4632-b350-eb3fa670018c-openshift-service-ca\") pod \"maas-api-5867c8d69-j8mzx\" (UID: \"da254e82-d544-4632-b350-eb3fa670018c\") " pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:06:39.900891 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:39.900785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da254e82-d544-4632-b350-eb3fa670018c-metrics-certs\") pod \"maas-api-5867c8d69-j8mzx\" (UID: \"da254e82-d544-4632-b350-eb3fa670018c\") " pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:06:39.900891 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:39.900847    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da254e82-d544-4632-b350-eb3fa670018c-maas-api-tls\") pod \"maas-api-5867c8d69-j8mzx\" (UID: \"da254e82-d544-4632-b350-eb3fa670018c\") " pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:06:40.001910 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.001877    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da254e82-d544-4632-b350-eb3fa670018c-openshift-service-ca\") pod \"maas-api-5867c8d69-j8mzx\" (UID: \"da254e82-d544-4632-b350-eb3fa670018c\") " pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:06:40.001910 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.001909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da254e82-d544-4632-b350-eb3fa670018c-metrics-certs\") pod \"maas-api-5867c8d69-j8mzx\" (UID: \"da254e82-d544-4632-b350-eb3fa670018c\") " pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:06:40.002120 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.001947    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da254e82-d544-4632-b350-eb3fa670018c-maas-api-tls\") pod \"maas-api-5867c8d69-j8mzx\" (UID: \"da254e82-d544-4632-b350-eb3fa670018c\") " pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:06:40.002120 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.001983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nc52\" (UniqueName: \"kubernetes.io/projected/da254e82-d544-4632-b350-eb3fa670018c-kube-api-access-9nc52\") pod \"maas-api-5867c8d69-j8mzx\" (UID: \"da254e82-d544-4632-b350-eb3fa670018c\") " pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:06:40.002741 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.002720    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da254e82-d544-4632-b350-eb3fa670018c-openshift-service-ca\") pod \"maas-api-5867c8d69-j8mzx\" (UID: \"da254e82-d544-4632-b350-eb3fa670018c\") " pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:06:40.004304 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.004281    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da254e82-d544-4632-b350-eb3fa670018c-maas-api-tls\") pod \"maas-api-5867c8d69-j8mzx\" (UID: \"da254e82-d544-4632-b350-eb3fa670018c\") " pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:06:40.004411 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.004364    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da254e82-d544-4632-b350-eb3fa670018c-metrics-certs\") pod \"maas-api-5867c8d69-j8mzx\" (UID: \"da254e82-d544-4632-b350-eb3fa670018c\") " pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:06:40.009593 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.009570    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nc52\" (UniqueName: \"kubernetes.io/projected/da254e82-d544-4632-b350-eb3fa670018c-kube-api-access-9nc52\") pod \"maas-api-5867c8d69-j8mzx\" (UID: \"da254e82-d544-4632-b350-eb3fa670018c\") " pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:06:40.073972 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.073950    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:06:40.290357 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.290326    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-kvhz2"]
Sep 10 21:06:40.293874 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.293859    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-kvhz2"
Sep 10 21:06:40.296100 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.296078    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 21:06:40.296214 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.296124    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-zlx7n\""
Sep 10 21:06:40.304504 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.304479    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-kvhz2"]
Sep 10 21:06:40.346450 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.346396    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"]
Sep 10 21:06:40.348456 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:06:40.348431    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda254e82_d544_4632_b350_eb3fa670018c.slice/crio-2eabb1fb358045b361777dbc97c15eede83840e4d31d2b30cdee3e18d7908204 WatchSource:0}: Error finding container 2eabb1fb358045b361777dbc97c15eede83840e4d31d2b30cdee3e18d7908204: Status 404 returned error can't find the container with id 2eabb1fb358045b361777dbc97c15eede83840e4d31d2b30cdee3e18d7908204
Sep 10 21:06:40.405496 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.405473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm5s9\" (UniqueName: \"kubernetes.io/projected/ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f-kube-api-access-dm5s9\") pod \"payload-processing-5b54b7fbb7-kvhz2\" (UID: \"ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kvhz2"
Sep 10 21:06:40.405625 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.405531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-kvhz2\" (UID: \"ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kvhz2"
Sep 10 21:06:40.506886 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.506855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm5s9\" (UniqueName: \"kubernetes.io/projected/ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f-kube-api-access-dm5s9\") pod \"payload-processing-5b54b7fbb7-kvhz2\" (UID: \"ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kvhz2"
Sep 10 21:06:40.507003 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.506912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-kvhz2\" (UID: \"ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kvhz2"
Sep 10 21:06:40.507561 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.507544    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-kvhz2\" (UID: \"ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kvhz2"
Sep 10 21:06:40.515819 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.515793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm5s9\" (UniqueName: \"kubernetes.io/projected/ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f-kube-api-access-dm5s9\") pod \"payload-processing-5b54b7fbb7-kvhz2\" (UID: \"ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kvhz2"
Sep 10 21:06:40.604364 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.604300    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-kvhz2"
Sep 10 21:06:40.873802 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:40.873748    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-kvhz2"]
Sep 10 21:06:40.875832 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:06:40.875803    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc0ba86_c132_406c_b14a_2cdc8d3a3c0f.slice/crio-ec270d7376704bc22b3c273a2bd26db448bc48e13889ee1d57291dfd4072db8a WatchSource:0}: Error finding container ec270d7376704bc22b3c273a2bd26db448bc48e13889ee1d57291dfd4072db8a: Status 404 returned error can't find the container with id ec270d7376704bc22b3c273a2bd26db448bc48e13889ee1d57291dfd4072db8a
Sep 10 21:06:41.084526 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:41.084489    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-kvhz2" event={"ID":"ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f","Type":"ContainerStarted","Data":"ec270d7376704bc22b3c273a2bd26db448bc48e13889ee1d57291dfd4072db8a"}
Sep 10 21:06:41.086071 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:41.086042    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx" event={"ID":"da254e82-d544-4632-b350-eb3fa670018c","Type":"ContainerStarted","Data":"2eabb1fb358045b361777dbc97c15eede83840e4d31d2b30cdee3e18d7908204"}
Sep 10 21:06:44.102497 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:44.102456    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx" event={"ID":"da254e82-d544-4632-b350-eb3fa670018c","Type":"ContainerStarted","Data":"acc9669f96daa23dd08f3ec4a85b52fc8325513b455fdb8a41ed1a5455091cf7"}
Sep 10 21:06:44.102964 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:44.102588    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:06:44.104095 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:44.104071    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-kvhz2" event={"ID":"ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f","Type":"ContainerStarted","Data":"11899b44a63f2bd8a35a307caec59644ccd2241ecbcf42e43530f6ff3bc79715"}
Sep 10 21:06:44.104236 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:44.104187    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-kvhz2"
Sep 10 21:06:44.121259 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:44.121153    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx" podStartSLOduration=1.5990550030000001 podStartE2EDuration="5.121138072s" totalImagesPullingTime="3.522083069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:44.119906052 +0000 UTC m=+545.095566114" watchObservedRunningTime="2026-09-10 21:06:44.121138072 +0000 UTC m=+545.096798117"
Sep 10 21:06:44.136938 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:44.136882    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-kvhz2" podStartSLOduration=1.097879773 podStartE2EDuration="4.136862249s" totalImagesPullingTime="3.038982476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:44.135790344 +0000 UTC m=+545.111450404" watchObservedRunningTime="2026-09-10 21:06:44.136862249 +0000 UTC m=+545.112522298"
Sep 10 21:06:50.114979 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:50.114951    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:06:55.111385 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:06:55.111350    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-kvhz2"
Sep 10 21:07:44.797679 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:44.797647    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"]
Sep 10 21:07:44.801033 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:44.801016    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:44.803177 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:44.803158    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 21:07:44.803355 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:44.803337    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mbf94\""
Sep 10 21:07:44.803965 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:44.803949    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 21:07:44.804043 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:44.804027    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 21:07:44.811251 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:44.811229    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"]
Sep 10 21:07:44.960255 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:44.960226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf057229-8150-4530-98e4-78e2844478ca-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:44.960449 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:44.960262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf057229-8150-4530-98e4-78e2844478ca-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:44.960449 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:44.960347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mhbn\" (UniqueName: \"kubernetes.io/projected/bf057229-8150-4530-98e4-78e2844478ca-kube-api-access-4mhbn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:44.960449 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:44.960386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf057229-8150-4530-98e4-78e2844478ca-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:44.960610 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:44.960454    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf057229-8150-4530-98e4-78e2844478ca-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:44.960610 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:44.960509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf057229-8150-4530-98e4-78e2844478ca-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:45.061512 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:45.061441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf057229-8150-4530-98e4-78e2844478ca-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:45.061512 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:45.061488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf057229-8150-4530-98e4-78e2844478ca-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:45.061697 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:45.061522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf057229-8150-4530-98e4-78e2844478ca-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:45.061697 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:45.061543    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf057229-8150-4530-98e4-78e2844478ca-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:45.061697 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:45.061570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhbn\" (UniqueName: \"kubernetes.io/projected/bf057229-8150-4530-98e4-78e2844478ca-kube-api-access-4mhbn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:45.061697 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:45.061588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf057229-8150-4530-98e4-78e2844478ca-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:45.061915 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:45.061892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf057229-8150-4530-98e4-78e2844478ca-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:45.061990 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:45.061961    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf057229-8150-4530-98e4-78e2844478ca-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:45.062026 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:45.061979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf057229-8150-4530-98e4-78e2844478ca-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:45.063790 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:45.063768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf057229-8150-4530-98e4-78e2844478ca-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:45.063969 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:45.063951    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf057229-8150-4530-98e4-78e2844478ca-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:45.071315 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:45.071294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhbn\" (UniqueName: \"kubernetes.io/projected/bf057229-8150-4530-98e4-78e2844478ca-kube-api-access-4mhbn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j\" (UID: \"bf057229-8150-4530-98e4-78e2844478ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:45.112592 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:45.112571    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:45.381912 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:45.381828    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"]
Sep 10 21:07:45.384922 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:07:45.384894    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf057229_8150_4530_98e4_78e2844478ca.slice/crio-fcc7ad283863218361adfadb1367bc7d4ec2e0c507352ae6cf4d99109af11457 WatchSource:0}: Error finding container fcc7ad283863218361adfadb1367bc7d4ec2e0c507352ae6cf4d99109af11457: Status 404 returned error can't find the container with id fcc7ad283863218361adfadb1367bc7d4ec2e0c507352ae6cf4d99109af11457
Sep 10 21:07:45.386118 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:45.386103    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:07:46.388637 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:46.388588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j" event={"ID":"bf057229-8150-4530-98e4-78e2844478ca","Type":"ContainerStarted","Data":"fcc7ad283863218361adfadb1367bc7d4ec2e0c507352ae6cf4d99109af11457"}
Sep 10 21:07:47.394411 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:47.394371    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j" event={"ID":"bf057229-8150-4530-98e4-78e2844478ca","Type":"ContainerStarted","Data":"add8e8a929deabd46b34fb29b7511b0f3391026ef5eaacb3d727297e81eef12a"}
Sep 10 21:07:47.412909 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:47.412859    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j" podStartSLOduration=2.398315189 podStartE2EDuration="3.41284343s" totalImagesPullingTime="1.014528241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:47.411469567 +0000 UTC m=+608.387129616" watchObservedRunningTime="2026-09-10 21:07:47.41284343 +0000 UTC m=+608.388503476"
Sep 10 21:07:55.112776 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:55.112745    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:55.113165 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:55.112805    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:55.124895 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:55.124871    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:55.443336 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:55.443260    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j"
Sep 10 21:07:56.601867 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.601838    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"]
Sep 10 21:07:56.605376 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.605359    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.607626 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.607600    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 21:07:56.614147 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.614127    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"]
Sep 10 21:07:56.664930 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.664882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lql9r\" (UniqueName: \"kubernetes.io/projected/60e26963-0401-4190-8e14-c995cec100e9-kube-api-access-lql9r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.664930 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.664931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e26963-0401-4190-8e14-c995cec100e9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.665167 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.665035    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e26963-0401-4190-8e14-c995cec100e9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.665235 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.665169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e26963-0401-4190-8e14-c995cec100e9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.665235 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.665219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e26963-0401-4190-8e14-c995cec100e9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.665366 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.665249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e26963-0401-4190-8e14-c995cec100e9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.765893 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.765865    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e26963-0401-4190-8e14-c995cec100e9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.765893 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.765895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e26963-0401-4190-8e14-c995cec100e9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.766107 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.765911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e26963-0401-4190-8e14-c995cec100e9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.766107 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.765939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lql9r\" (UniqueName: \"kubernetes.io/projected/60e26963-0401-4190-8e14-c995cec100e9-kube-api-access-lql9r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.766107 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.765956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e26963-0401-4190-8e14-c995cec100e9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.766107 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.766014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e26963-0401-4190-8e14-c995cec100e9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.766322 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.766306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e26963-0401-4190-8e14-c995cec100e9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.766411 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.766390    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e26963-0401-4190-8e14-c995cec100e9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.766458 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.766438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e26963-0401-4190-8e14-c995cec100e9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.768149 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.768129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e26963-0401-4190-8e14-c995cec100e9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.768410 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.768391    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e26963-0401-4190-8e14-c995cec100e9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.773965 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.773944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lql9r\" (UniqueName: \"kubernetes.io/projected/60e26963-0401-4190-8e14-c995cec100e9-kube-api-access-lql9r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v\" (UID: \"60e26963-0401-4190-8e14-c995cec100e9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:56.917098 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:56.917016    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:07:57.201092 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:57.201014    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"]
Sep 10 21:07:57.203379 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:07:57.203348    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e26963_0401_4190_8e14_c995cec100e9.slice/crio-bbf9e652b17faa1faadecd973a3ae73b08cef7312669175e0da1136770997ab8 WatchSource:0}: Error finding container bbf9e652b17faa1faadecd973a3ae73b08cef7312669175e0da1136770997ab8: Status 404 returned error can't find the container with id bbf9e652b17faa1faadecd973a3ae73b08cef7312669175e0da1136770997ab8
Sep 10 21:07:57.441227 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:57.441185    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v" event={"ID":"60e26963-0401-4190-8e14-c995cec100e9","Type":"ContainerStarted","Data":"bbf9e652b17faa1faadecd973a3ae73b08cef7312669175e0da1136770997ab8"}
Sep 10 21:07:58.447560 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:58.447524    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v" event={"ID":"60e26963-0401-4190-8e14-c995cec100e9","Type":"ContainerStarted","Data":"7ad5dcc5de04b9f8510a0dc49aa8187e458c2c6e53e6efd6022ac39078adbbd7"}
Sep 10 21:07:58.465913 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:07:58.465805    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v" podStartSLOduration=2.238480603 podStartE2EDuration="2.465785834s" totalImagesPullingTime="227.305231ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:58.465479847 +0000 UTC m=+619.441139895" watchObservedRunningTime="2026-09-10 21:07:58.465785834 +0000 UTC m=+619.441445882"
Sep 10 21:08:06.917913 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:06.917831    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:08:06.917913 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:06.917904    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:08:06.930549 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:06.930524    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:08:07.500434 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:07.500405    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v"
Sep 10 21:08:14.097923 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.097887    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"]
Sep 10 21:08:14.104181 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.104156    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.106291 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.106266    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 10 21:08:14.110453 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.110427    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"]
Sep 10 21:08:14.219305 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.219274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.219477 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.219322    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.219477 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.219349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mmkw\" (UniqueName: \"kubernetes.io/projected/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-kube-api-access-2mmkw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.219477 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.219400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.219477 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.219451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.219477 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.219477    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.320588 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.320559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.320734 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.320593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.320734 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.320627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.320734 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.320652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.320734 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.320681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mmkw\" (UniqueName: \"kubernetes.io/projected/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-kube-api-access-2mmkw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.320734 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.320708    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.320982 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.320961    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.321092 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.321069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.321170 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.321103    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.323044 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.323020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.323296 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.323279    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.328718 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.328701    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mmkw\" (UniqueName: \"kubernetes.io/projected/f15ece67-3fa7-4f0e-9f49-83abb10cdcaa-kube-api-access-2mmkw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndhj8\" (UID: \"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.416171 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.416099    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:14.695798 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:14.695775    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"]
Sep 10 21:08:14.697665 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:08:14.697638    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15ece67_3fa7_4f0e_9f49_83abb10cdcaa.slice/crio-1646531888d5f72439678bdd1070de5c7ecb1d6be65ffe9f4a6698064383fb74 WatchSource:0}: Error finding container 1646531888d5f72439678bdd1070de5c7ecb1d6be65ffe9f4a6698064383fb74: Status 404 returned error can't find the container with id 1646531888d5f72439678bdd1070de5c7ecb1d6be65ffe9f4a6698064383fb74
Sep 10 21:08:15.531674 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:15.531644    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8" event={"ID":"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa","Type":"ContainerStarted","Data":"750ca88abc3c69ba5a6cc21788f9392fd8da7d55f2b546ca3a14151100975cd8"}
Sep 10 21:08:15.532108 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:15.531711    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8" event={"ID":"f15ece67-3fa7-4f0e-9f49-83abb10cdcaa","Type":"ContainerStarted","Data":"1646531888d5f72439678bdd1070de5c7ecb1d6be65ffe9f4a6698064383fb74"}
Sep 10 21:08:15.548024 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:15.547966    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8" podStartSLOduration=1.3328355219999999 podStartE2EDuration="1.547951213s" totalImagesPullingTime="215.115691ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:08:15.547021949 +0000 UTC m=+636.522682035" watchObservedRunningTime="2026-09-10 21:08:15.547951213 +0000 UTC m=+636.523611256"
Sep 10 21:08:24.416882 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:24.416842    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:24.417278 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:24.416914    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:24.429191 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:24.429164    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:24.588390 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:24.588362    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndhj8"
Sep 10 21:08:25.413092 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.413060    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"]
Sep 10 21:08:25.416385 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.416366    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.418612 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.418587    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 21:08:25.428030 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.428008    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"]
Sep 10 21:08:25.516924 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.516897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deca99e8-58d2-458f-9f16-9f51ce6f802e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.517073 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.516952    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deca99e8-58d2-458f-9f16-9f51ce6f802e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.517073 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.516978    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llbgn\" (UniqueName: \"kubernetes.io/projected/deca99e8-58d2-458f-9f16-9f51ce6f802e-kube-api-access-llbgn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.517073 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.517054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deca99e8-58d2-458f-9f16-9f51ce6f802e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.517179 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.517081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deca99e8-58d2-458f-9f16-9f51ce6f802e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.517179 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.517103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deca99e8-58d2-458f-9f16-9f51ce6f802e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.618299 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.618270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deca99e8-58d2-458f-9f16-9f51ce6f802e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.618440 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.618318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deca99e8-58d2-458f-9f16-9f51ce6f802e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.618440 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.618342    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llbgn\" (UniqueName: \"kubernetes.io/projected/deca99e8-58d2-458f-9f16-9f51ce6f802e-kube-api-access-llbgn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.618440 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.618372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deca99e8-58d2-458f-9f16-9f51ce6f802e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.618440 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.618387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deca99e8-58d2-458f-9f16-9f51ce6f802e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.618440 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.618410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deca99e8-58d2-458f-9f16-9f51ce6f802e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.618691 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.618671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deca99e8-58d2-458f-9f16-9f51ce6f802e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.618745 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.618724    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deca99e8-58d2-458f-9f16-9f51ce6f802e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.618791 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.618762    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deca99e8-58d2-458f-9f16-9f51ce6f802e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.620590 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.620569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deca99e8-58d2-458f-9f16-9f51ce6f802e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.620852 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.620835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deca99e8-58d2-458f-9f16-9f51ce6f802e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.625748 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.625729    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llbgn\" (UniqueName: \"kubernetes.io/projected/deca99e8-58d2-458f-9f16-9f51ce6f802e-kube-api-access-llbgn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h\" (UID: \"deca99e8-58d2-458f-9f16-9f51ce6f802e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.726308 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.726248    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:25.996281 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:25.996255    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"]
Sep 10 21:08:25.998388 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:08:25.998359    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddeca99e8_58d2_458f_9f16_9f51ce6f802e.slice/crio-25272054932f2a493974fae9dac080fd4bd8249ea0c5f2dbe391fb6dd5bc8f96 WatchSource:0}: Error finding container 25272054932f2a493974fae9dac080fd4bd8249ea0c5f2dbe391fb6dd5bc8f96: Status 404 returned error can't find the container with id 25272054932f2a493974fae9dac080fd4bd8249ea0c5f2dbe391fb6dd5bc8f96
Sep 10 21:08:26.582545 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:26.582513    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h" event={"ID":"deca99e8-58d2-458f-9f16-9f51ce6f802e","Type":"ContainerStarted","Data":"bee8d6554be1533e3815e6e58277b66acd8c3090190c5b3714191c02aa720cd3"}
Sep 10 21:08:26.582911 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:26.582552    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h" event={"ID":"deca99e8-58d2-458f-9f16-9f51ce6f802e","Type":"ContainerStarted","Data":"25272054932f2a493974fae9dac080fd4bd8249ea0c5f2dbe391fb6dd5bc8f96"}
Sep 10 21:08:26.600082 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:26.600037    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h" podStartSLOduration=1.361405067 podStartE2EDuration="1.600017406s" totalImagesPullingTime="238.612339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:08:26.598805195 +0000 UTC m=+647.574465241" watchObservedRunningTime="2026-09-10 21:08:26.600017406 +0000 UTC m=+647.575677452"
Sep 10 21:08:35.727130 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:35.727097    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:35.727540 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:35.727145    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:35.739578 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:35.739549    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:08:36.638291 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:08:36.638262    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h"
Sep 10 21:09:47.341186 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:47.341100    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"]
Sep 10 21:09:47.345137 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:47.345112    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:09:47.347592 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:47.347570    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-47f62f-serving-cert\""
Sep 10 21:09:47.359591 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:47.359564    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"]
Sep 10 21:09:47.450475 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:47.450432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93023191-175a-4709-9855-e154821a741e-maas-api-tls\") pod \"maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf\" (UID: \"93023191-175a-4709-9855-e154821a741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:09:47.450661 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:47.450479    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93023191-175a-4709-9855-e154821a741e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf\" (UID: \"93023191-175a-4709-9855-e154821a741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:09:47.450661 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:47.450542    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn7lq\" (UniqueName: \"kubernetes.io/projected/93023191-175a-4709-9855-e154821a741e-kube-api-access-zn7lq\") pod \"maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf\" (UID: \"93023191-175a-4709-9855-e154821a741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:09:47.450661 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:47.450570    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93023191-175a-4709-9855-e154821a741e-metrics-certs\") pod \"maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf\" (UID: \"93023191-175a-4709-9855-e154821a741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:09:47.551460 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:47.551423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93023191-175a-4709-9855-e154821a741e-maas-api-tls\") pod \"maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf\" (UID: \"93023191-175a-4709-9855-e154821a741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:09:47.551460 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:47.551469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93023191-175a-4709-9855-e154821a741e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf\" (UID: \"93023191-175a-4709-9855-e154821a741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:09:47.551911 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:47.551515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn7lq\" (UniqueName: \"kubernetes.io/projected/93023191-175a-4709-9855-e154821a741e-kube-api-access-zn7lq\") pod \"maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf\" (UID: \"93023191-175a-4709-9855-e154821a741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:09:47.551911 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:47.551543    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93023191-175a-4709-9855-e154821a741e-metrics-certs\") pod \"maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf\" (UID: \"93023191-175a-4709-9855-e154821a741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:09:47.551911 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:09:47.551596    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-serving-cert: secret "maas-api-e2e-inf-a-47f62f-serving-cert" not found
Sep 10 21:09:47.551911 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:09:47.551690    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93023191-175a-4709-9855-e154821a741e-maas-api-tls podName:93023191-175a-4709-9855-e154821a741e nodeName:}" failed. No retries permitted until 2026-09-10 21:09:48.051668967 +0000 UTC m=+729.027329005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93023191-175a-4709-9855-e154821a741e-maas-api-tls") pod "maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf" (UID: "93023191-175a-4709-9855-e154821a741e") : secret "maas-api-e2e-inf-a-47f62f-serving-cert" not found
Sep 10 21:09:47.552257 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:47.552232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93023191-175a-4709-9855-e154821a741e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf\" (UID: \"93023191-175a-4709-9855-e154821a741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:09:47.554176 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:47.554146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93023191-175a-4709-9855-e154821a741e-metrics-certs\") pod \"maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf\" (UID: \"93023191-175a-4709-9855-e154821a741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:09:47.563035 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:47.563010    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn7lq\" (UniqueName: \"kubernetes.io/projected/93023191-175a-4709-9855-e154821a741e-kube-api-access-zn7lq\") pod \"maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf\" (UID: \"93023191-175a-4709-9855-e154821a741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:09:48.056082 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:48.056038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93023191-175a-4709-9855-e154821a741e-maas-api-tls\") pod \"maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf\" (UID: \"93023191-175a-4709-9855-e154821a741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:09:48.058257 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:48.058231    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93023191-175a-4709-9855-e154821a741e-maas-api-tls\") pod \"maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf\" (UID: \"93023191-175a-4709-9855-e154821a741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:09:48.256009 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:48.255974    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:09:48.531603 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:48.531575    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"]
Sep 10 21:09:48.534649 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:09:48.534619    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93023191_175a_4709_9855_e154821a741e.slice/crio-c4abcb3a270c1dd61a981b03797be1b0aae5dc0ea6759d8305857ead7aca8bce WatchSource:0}: Error finding container c4abcb3a270c1dd61a981b03797be1b0aae5dc0ea6759d8305857ead7aca8bce: Status 404 returned error can't find the container with id c4abcb3a270c1dd61a981b03797be1b0aae5dc0ea6759d8305857ead7aca8bce
Sep 10 21:09:48.976411 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:48.976389    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf" event={"ID":"93023191-175a-4709-9855-e154821a741e","Type":"ContainerStarted","Data":"c4abcb3a270c1dd61a981b03797be1b0aae5dc0ea6759d8305857ead7aca8bce"}
Sep 10 21:09:49.983042 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:49.983003    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf" event={"ID":"93023191-175a-4709-9855-e154821a741e","Type":"ContainerStarted","Data":"a997db38d9a89b915c69ee7baf0f53f4fc5f062f7b4ce2cc27b129dcc11b3b31"}
Sep 10 21:09:49.983450 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:49.983159    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:09:50.003314 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:50.003273    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf" podStartSLOduration=2.5809399600000003 podStartE2EDuration="3.003258541s" totalImagesPullingTime="422.318581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:50.002734042 +0000 UTC m=+730.978394093" watchObservedRunningTime="2026-09-10 21:09:50.003258541 +0000 UTC m=+730.978918602"
Sep 10 21:09:55.817320 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:55.817289    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"]
Sep 10 21:09:55.820667 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:55.820646    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:09:55.823602 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:55.823581    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1a398fc7-serving-cert\""
Sep 10 21:09:55.835523 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:55.835502    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"]
Sep 10 21:09:55.919571 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:55.919530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-metrics-certs\") pod \"maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth\" (UID: \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:09:55.919740 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:55.919592    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s2p9\" (UniqueName: \"kubernetes.io/projected/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-kube-api-access-9s2p9\") pod \"maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth\" (UID: \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:09:55.919740 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:55.919693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-openshift-service-ca\") pod \"maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth\" (UID: \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:09:55.919815 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:55.919765    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-maas-api-tls\") pod \"maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth\" (UID: \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:09:55.994840 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:55.994811    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:09:56.020871 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:56.020839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-openshift-service-ca\") pod \"maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth\" (UID: \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:09:56.021052 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:56.020896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-maas-api-tls\") pod \"maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth\" (UID: \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:09:56.021052 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:56.020948    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-metrics-certs\") pod \"maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth\" (UID: \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:09:56.021052 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:56.020997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s2p9\" (UniqueName: \"kubernetes.io/projected/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-kube-api-access-9s2p9\") pod \"maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth\" (UID: \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:09:56.021240 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:09:56.021172    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-serving-cert: secret "maas-api-e2e-ait-1a398fc7-serving-cert" not found
Sep 10 21:09:56.021309 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:09:56.021264    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-maas-api-tls podName:b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9 nodeName:}" failed. No retries permitted until 2026-09-10 21:09:56.52124469 +0000 UTC m=+737.496904723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-maas-api-tls") pod "maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth" (UID: "b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9") : secret "maas-api-e2e-ait-1a398fc7-serving-cert" not found
Sep 10 21:09:56.021858 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:56.021835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-openshift-service-ca\") pod \"maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth\" (UID: \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:09:56.023740 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:56.023720    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-metrics-certs\") pod \"maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth\" (UID: \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:09:56.035239 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:56.035197    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s2p9\" (UniqueName: \"kubernetes.io/projected/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-kube-api-access-9s2p9\") pod \"maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth\" (UID: \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:09:56.526804 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:56.526773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-maas-api-tls\") pod \"maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth\" (UID: \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:09:56.528996 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:56.528974    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-maas-api-tls\") pod \"maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth\" (UID: \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:09:56.730832 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:56.730798    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:09:57.008224 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:57.008173    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"]
Sep 10 21:09:57.009501 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:09:57.009475    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb353c26f_e0b1_4d3a_b9cf_45bfbebfedd9.slice/crio-c9a6d9c538150195bb0c6d54ff57f1089b67b5c9d2aad6f37bb9c35e59bae383 WatchSource:0}: Error finding container c9a6d9c538150195bb0c6d54ff57f1089b67b5c9d2aad6f37bb9c35e59bae383: Status 404 returned error can't find the container with id c9a6d9c538150195bb0c6d54ff57f1089b67b5c9d2aad6f37bb9c35e59bae383
Sep 10 21:09:57.017716 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:57.017693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth" event={"ID":"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9","Type":"ContainerStarted","Data":"c9a6d9c538150195bb0c6d54ff57f1089b67b5c9d2aad6f37bb9c35e59bae383"}
Sep 10 21:09:58.026023 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:58.025985    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth" event={"ID":"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9","Type":"ContainerStarted","Data":"1d90d2dfe5548d3286049731141713f5b00bf7f09c5362cf215c7df559b4bb44"}
Sep 10 21:09:58.026512 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:58.026152    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:09:58.063158 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:09:58.063114    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth" podStartSLOduration=2.739170472 podStartE2EDuration="3.063100227s" totalImagesPullingTime="323.929755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:58.061758796 +0000 UTC m=+739.037418844" watchObservedRunningTime="2026-09-10 21:09:58.063100227 +0000 UTC m=+739.038760273"
Sep 10 21:10:04.040217 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:04.040174    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:10:12.921801 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:12.921764    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"]
Sep 10 21:10:12.925695 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:12.925678    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:12.932808 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:12.932788    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-869961db-kserve-self-signed-certs\""
Sep 10 21:10:12.960433 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:12.960408    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"]
Sep 10 21:10:12.997278 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:12.997249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-home\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:12.997390 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:12.997302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-tmp-dir\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:12.997390 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:12.997332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-dshm\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:12.997390 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:12.997382    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs8x5\" (UniqueName: \"kubernetes.io/projected/3488c548-8089-4422-a20d-67d8505f2b82-kube-api-access-bs8x5\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:12.997500 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:12.997419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-model-cache\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:12.997500 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:12.997447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3488c548-8089-4422-a20d-67d8505f2b82-tls-certs\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:13.097967 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:13.097940    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-model-cache\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:13.098144 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:13.097981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3488c548-8089-4422-a20d-67d8505f2b82-tls-certs\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:13.098144 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:13.098031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-home\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:13.098144 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:13.098070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-tmp-dir\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:13.098144 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:13.098090    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-dshm\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:13.098144 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:13.098132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs8x5\" (UniqueName: \"kubernetes.io/projected/3488c548-8089-4422-a20d-67d8505f2b82-kube-api-access-bs8x5\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:13.098449 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:13.098421    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-model-cache\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:13.098562 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:13.098541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-tmp-dir\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:13.098622 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:13.098610    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-home\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:13.100390 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:13.100370    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-dshm\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:13.100836 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:13.100819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3488c548-8089-4422-a20d-67d8505f2b82-tls-certs\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:13.106133 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:13.106116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs8x5\" (UniqueName: \"kubernetes.io/projected/3488c548-8089-4422-a20d-67d8505f2b82-kube-api-access-bs8x5\") pod \"e2e-conflict-a-869961db-kserve-6997798744-f6kwl\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") " pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:13.234705 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:13.234625    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:13.532697 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:13.532667    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"]
Sep 10 21:10:13.534516 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:10:13.534485    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3488c548_8089_4422_a20d_67d8505f2b82.slice/crio-8e637fb5f4b7c0437834b5b0ccf53fa8a2fb95cd557661c9cbf37754bc874c84 WatchSource:0}: Error finding container 8e637fb5f4b7c0437834b5b0ccf53fa8a2fb95cd557661c9cbf37754bc874c84: Status 404 returned error can't find the container with id 8e637fb5f4b7c0437834b5b0ccf53fa8a2fb95cd557661c9cbf37754bc874c84
Sep 10 21:10:14.110936 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:14.110901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl" event={"ID":"3488c548-8089-4422-a20d-67d8505f2b82","Type":"ContainerStarted","Data":"79ad9354ba383ef0dfe3909922cf68326a2d6f64326cc9566f1028a50d2bd660"}
Sep 10 21:10:14.111322 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:14.110945    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl" event={"ID":"3488c548-8089-4422-a20d-67d8505f2b82","Type":"ContainerStarted","Data":"8e637fb5f4b7c0437834b5b0ccf53fa8a2fb95cd557661c9cbf37754bc874c84"}
Sep 10 21:10:14.129742 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:14.129691    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl" podStartSLOduration=2.1296775390000002 podStartE2EDuration="2.129677539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:14.127892918 +0000 UTC m=+755.103552966" watchObservedRunningTime="2026-09-10 21:10:14.129677539 +0000 UTC m=+755.105337585"
Sep 10 21:10:16.652083 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:16.652043    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"]
Sep 10 21:10:16.656089 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:16.656067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:10:16.658428 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:16.658411    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-000295-serving-cert\""
Sep 10 21:10:16.672853 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:16.672829    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"]
Sep 10 21:10:16.731943 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:16.731908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/974f55b7-61ea-4f59-8778-306171335bd7-maas-api-tls\") pod \"maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr\" (UID: \"974f55b7-61ea-4f59-8778-306171335bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:10:16.732074 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:16.731972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/974f55b7-61ea-4f59-8778-306171335bd7-metrics-certs\") pod \"maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr\" (UID: \"974f55b7-61ea-4f59-8778-306171335bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:10:16.732124 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:16.732088    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz9xl\" (UniqueName: \"kubernetes.io/projected/974f55b7-61ea-4f59-8778-306171335bd7-kube-api-access-xz9xl\") pod \"maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr\" (UID: \"974f55b7-61ea-4f59-8778-306171335bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:10:16.732124 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:16.732117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/974f55b7-61ea-4f59-8778-306171335bd7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr\" (UID: \"974f55b7-61ea-4f59-8778-306171335bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:10:16.833245 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:16.833191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/974f55b7-61ea-4f59-8778-306171335bd7-maas-api-tls\") pod \"maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr\" (UID: \"974f55b7-61ea-4f59-8778-306171335bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:10:16.833431 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:16.833274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/974f55b7-61ea-4f59-8778-306171335bd7-metrics-certs\") pod \"maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr\" (UID: \"974f55b7-61ea-4f59-8778-306171335bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:10:16.833431 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:16.833336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz9xl\" (UniqueName: \"kubernetes.io/projected/974f55b7-61ea-4f59-8778-306171335bd7-kube-api-access-xz9xl\") pod \"maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr\" (UID: \"974f55b7-61ea-4f59-8778-306171335bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:10:16.833558 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:16.833517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/974f55b7-61ea-4f59-8778-306171335bd7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr\" (UID: \"974f55b7-61ea-4f59-8778-306171335bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:10:16.834087 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:16.834064    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/974f55b7-61ea-4f59-8778-306171335bd7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr\" (UID: \"974f55b7-61ea-4f59-8778-306171335bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:10:16.835747 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:16.835721    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/974f55b7-61ea-4f59-8778-306171335bd7-maas-api-tls\") pod \"maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr\" (UID: \"974f55b7-61ea-4f59-8778-306171335bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:10:16.835864 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:16.835800    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/974f55b7-61ea-4f59-8778-306171335bd7-metrics-certs\") pod \"maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr\" (UID: \"974f55b7-61ea-4f59-8778-306171335bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:10:16.842072 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:16.842049    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz9xl\" (UniqueName: \"kubernetes.io/projected/974f55b7-61ea-4f59-8778-306171335bd7-kube-api-access-xz9xl\") pod \"maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr\" (UID: \"974f55b7-61ea-4f59-8778-306171335bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:10:16.969017 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:16.968949    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:10:17.261900 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:17.261817    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"]
Sep 10 21:10:17.265932 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:10:17.265906    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974f55b7_61ea_4f59_8778_306171335bd7.slice/crio-526c2864e9d5afa1a0b56086513abd5425f565763e33ba1cb852421aab331bf3 WatchSource:0}: Error finding container 526c2864e9d5afa1a0b56086513abd5425f565763e33ba1cb852421aab331bf3: Status 404 returned error can't find the container with id 526c2864e9d5afa1a0b56086513abd5425f565763e33ba1cb852421aab331bf3
Sep 10 21:10:18.132648 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:18.132616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr" event={"ID":"974f55b7-61ea-4f59-8778-306171335bd7","Type":"ContainerStarted","Data":"f0f06cc38bd33e37bd3b29e2976eb887859b992b5b0f8045def0f9e7880f389d"}
Sep 10 21:10:18.133169 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:18.132666    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr" event={"ID":"974f55b7-61ea-4f59-8778-306171335bd7","Type":"ContainerStarted","Data":"526c2864e9d5afa1a0b56086513abd5425f565763e33ba1cb852421aab331bf3"}
Sep 10 21:10:18.133169 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:18.132738    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:10:18.159996 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:18.159947    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr" podStartSLOduration=1.8030618010000001 podStartE2EDuration="2.1599313s" totalImagesPullingTime="356.869499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:18.15840948 +0000 UTC m=+759.134069527" watchObservedRunningTime="2026-09-10 21:10:18.1599313 +0000 UTC m=+759.135591346"
Sep 10 21:10:23.235226 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:23.235183    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:23.235651 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:23.235241    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:23.248001 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:23.247982    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:24.142491 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:24.142459    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:10:24.171907 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:24.171879    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:10:36.949340 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:36.949299    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"]
Sep 10 21:10:36.949877 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:36.949491    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth" podUID="b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9" containerName="maas-api" containerID="cri-o://1d90d2dfe5548d3286049731141713f5b00bf7f09c5362cf215c7df559b4bb44" gracePeriod=30
Sep 10 21:10:37.210320 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.210258    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:10:37.229109 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.229075    2581 generic.go:356] "Generic (PLEG): container finished" podID="b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9" containerID="1d90d2dfe5548d3286049731141713f5b00bf7f09c5362cf215c7df559b4bb44" exitCode=0
Sep 10 21:10:37.229277 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.229258    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth" event={"ID":"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9","Type":"ContainerDied","Data":"1d90d2dfe5548d3286049731141713f5b00bf7f09c5362cf215c7df559b4bb44"}
Sep 10 21:10:37.229337 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.229309    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth" event={"ID":"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9","Type":"ContainerDied","Data":"c9a6d9c538150195bb0c6d54ff57f1089b67b5c9d2aad6f37bb9c35e59bae383"}
Sep 10 21:10:37.229385 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.229338    2581 scope.go:117] "RemoveContainer" containerID="1d90d2dfe5548d3286049731141713f5b00bf7f09c5362cf215c7df559b4bb44"
Sep 10 21:10:37.229507 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.229489    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"
Sep 10 21:10:37.238993 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.238975    2581 scope.go:117] "RemoveContainer" containerID="1d90d2dfe5548d3286049731141713f5b00bf7f09c5362cf215c7df559b4bb44"
Sep 10 21:10:37.239341 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:10:37.239315    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d90d2dfe5548d3286049731141713f5b00bf7f09c5362cf215c7df559b4bb44\": container with ID starting with 1d90d2dfe5548d3286049731141713f5b00bf7f09c5362cf215c7df559b4bb44 not found: ID does not exist" containerID="1d90d2dfe5548d3286049731141713f5b00bf7f09c5362cf215c7df559b4bb44"
Sep 10 21:10:37.239428 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.239355    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d90d2dfe5548d3286049731141713f5b00bf7f09c5362cf215c7df559b4bb44"} err="failed to get container status \"1d90d2dfe5548d3286049731141713f5b00bf7f09c5362cf215c7df559b4bb44\": rpc error: code = NotFound desc = could not find container \"1d90d2dfe5548d3286049731141713f5b00bf7f09c5362cf215c7df559b4bb44\": container with ID starting with 1d90d2dfe5548d3286049731141713f5b00bf7f09c5362cf215c7df559b4bb44 not found: ID does not exist"
Sep 10 21:10:37.331264 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.331235    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s2p9\" (UniqueName: \"kubernetes.io/projected/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-kube-api-access-9s2p9\") pod \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\" (UID: \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\") "
Sep 10 21:10:37.331446 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.331282    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-maas-api-tls\") pod \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\" (UID: \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\") "
Sep 10 21:10:37.331446 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.331405    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-metrics-certs\") pod \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\" (UID: \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\") "
Sep 10 21:10:37.331446 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.331428    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-openshift-service-ca\") pod \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\" (UID: \"b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9\") "
Sep 10 21:10:37.331877 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.331844    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9" (UID: "b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:10:37.333493 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.333462    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9" (UID: "b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:10:37.333493 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.333478    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9" (UID: "b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:10:37.333630 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.333516    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-kube-api-access-9s2p9" (OuterVolumeSpecName: "kube-api-access-9s2p9") pod "b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9" (UID: "b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9"). InnerVolumeSpecName "kube-api-access-9s2p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:10:37.432224 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.432171    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-metrics-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:10:37.432224 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.432214    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-openshift-service-ca\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:10:37.432224 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.432230    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s2p9\" (UniqueName: \"kubernetes.io/projected/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-kube-api-access-9s2p9\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:10:37.432444 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.432239    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9-maas-api-tls\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:10:37.552348 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.552320    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"]
Sep 10 21:10:37.557326 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.557303    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a398fc7-845f49f4b-9dtth"]
Sep 10 21:10:37.573500 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:37.573474    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9" path="/var/lib/kubelet/pods/b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9/volumes"
Sep 10 21:10:41.318317 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.318282    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"]
Sep 10 21:10:41.318783 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.318765    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9" containerName="maas-api"
Sep 10 21:10:41.318783 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.318782    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9" containerName="maas-api"
Sep 10 21:10:41.318900 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.318888    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b353c26f-e0b1-4d3a-b9cf-45bfbebfedd9" containerName="maas-api"
Sep 10 21:10:41.323766 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.323743    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:10:41.326020 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.326002    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-a7bbcc15-serving-cert\""
Sep 10 21:10:41.333225 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.333179    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"]
Sep 10 21:10:41.367508 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.367485    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-metrics-certs\") pod \"maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx\" (UID: \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:10:41.367620 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.367540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-openshift-service-ca\") pod \"maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx\" (UID: \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:10:41.367665 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.367615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w97kf\" (UniqueName: \"kubernetes.io/projected/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-kube-api-access-w97kf\") pod \"maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx\" (UID: \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:10:41.367665 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.367643    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-maas-api-tls\") pod \"maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx\" (UID: \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:10:41.468799 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.468762    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w97kf\" (UniqueName: \"kubernetes.io/projected/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-kube-api-access-w97kf\") pod \"maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx\" (UID: \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:10:41.468799 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.468801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-maas-api-tls\") pod \"maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx\" (UID: \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:10:41.469034 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.468846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-metrics-certs\") pod \"maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx\" (UID: \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:10:41.469034 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.468886    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-openshift-service-ca\") pod \"maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx\" (UID: \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:10:41.469034 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:10:41.468975    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-serving-cert: secret "maas-api-e2e-migrate-a7bbcc15-serving-cert" not found
Sep 10 21:10:41.469187 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:10:41.469037    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-maas-api-tls podName:a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2 nodeName:}" failed. No retries permitted until 2026-09-10 21:10:41.969017953 +0000 UTC m=+782.944677993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-maas-api-tls") pod "maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx" (UID: "a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2") : secret "maas-api-e2e-migrate-a7bbcc15-serving-cert" not found
Sep 10 21:10:41.469501 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.469472    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-openshift-service-ca\") pod \"maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx\" (UID: \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:10:41.471491 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.471470    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-metrics-certs\") pod \"maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx\" (UID: \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:10:41.481432 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.481409    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w97kf\" (UniqueName: \"kubernetes.io/projected/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-kube-api-access-w97kf\") pod \"maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx\" (UID: \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:10:41.971871 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.971830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-maas-api-tls\") pod \"maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx\" (UID: \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:10:41.974149 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:41.974126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-maas-api-tls\") pod \"maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx\" (UID: \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:10:42.236470 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:42.236385    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:10:42.509473 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:42.509448    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"]
Sep 10 21:10:42.511670 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:10:42.511636    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85b3a22_0325_46c4_b2f1_9f0dfc6b6cc2.slice/crio-092c1e1ee3f96abec1bfd9d8642f3209e6f3a0d27f8120e2a65f9e770a2b4484 WatchSource:0}: Error finding container 092c1e1ee3f96abec1bfd9d8642f3209e6f3a0d27f8120e2a65f9e770a2b4484: Status 404 returned error can't find the container with id 092c1e1ee3f96abec1bfd9d8642f3209e6f3a0d27f8120e2a65f9e770a2b4484
Sep 10 21:10:43.261543 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:43.261469    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx" event={"ID":"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2","Type":"ContainerStarted","Data":"a16634b3ab0a2dd73038ba94167735afad9f7d12becc6cdf8a3fac7e653ef233"}
Sep 10 21:10:43.261543 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:43.261520    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx" event={"ID":"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2","Type":"ContainerStarted","Data":"092c1e1ee3f96abec1bfd9d8642f3209e6f3a0d27f8120e2a65f9e770a2b4484"}
Sep 10 21:10:43.261728 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:43.261576    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:10:43.282379 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:43.282328    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx" podStartSLOduration=1.877125447 podStartE2EDuration="2.282313726s" totalImagesPullingTime="405.188279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:43.278740191 +0000 UTC m=+784.254400249" watchObservedRunningTime="2026-09-10 21:10:43.282313726 +0000 UTC m=+784.257973773"
Sep 10 21:10:49.274475 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:10:49.274440    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:11:12.691648 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:12.691608    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"]
Sep 10 21:11:12.692161 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:12.691863    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx" podUID="a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2" containerName="maas-api" containerID="cri-o://a16634b3ab0a2dd73038ba94167735afad9f7d12becc6cdf8a3fac7e653ef233" gracePeriod=30
Sep 10 21:11:12.930498 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:12.930476    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:11:13.074014 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.073972    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-metrics-certs\") pod \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\" (UID: \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\") "
Sep 10 21:11:13.074212 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.074046    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-openshift-service-ca\") pod \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\" (UID: \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\") "
Sep 10 21:11:13.074212 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.074147    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w97kf\" (UniqueName: \"kubernetes.io/projected/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-kube-api-access-w97kf\") pod \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\" (UID: \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\") "
Sep 10 21:11:13.074356 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.074229    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-maas-api-tls\") pod \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\" (UID: \"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2\") "
Sep 10 21:11:13.074685 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.074437    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2" (UID: "a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:11:13.074685 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.074654    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-openshift-service-ca\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:11:13.076993 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.076891    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2" (UID: "a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:11:13.076993 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.076957    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-kube-api-access-w97kf" (OuterVolumeSpecName: "kube-api-access-w97kf") pod "a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2" (UID: "a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2"). InnerVolumeSpecName "kube-api-access-w97kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:11:13.076993 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.076973    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2" (UID: "a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:11:13.175146 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.175115    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w97kf\" (UniqueName: \"kubernetes.io/projected/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-kube-api-access-w97kf\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:11:13.175146 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.175141    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-maas-api-tls\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:11:13.175146 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.175151    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2-metrics-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:11:13.416366 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.416283    2581 generic.go:356] "Generic (PLEG): container finished" podID="a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2" containerID="a16634b3ab0a2dd73038ba94167735afad9f7d12becc6cdf8a3fac7e653ef233" exitCode=0
Sep 10 21:11:13.416366 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.416325    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"
Sep 10 21:11:13.416535 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.416370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx" event={"ID":"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2","Type":"ContainerDied","Data":"a16634b3ab0a2dd73038ba94167735afad9f7d12becc6cdf8a3fac7e653ef233"}
Sep 10 21:11:13.416535 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.416425    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx" event={"ID":"a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2","Type":"ContainerDied","Data":"092c1e1ee3f96abec1bfd9d8642f3209e6f3a0d27f8120e2a65f9e770a2b4484"}
Sep 10 21:11:13.416535 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.416449    2581 scope.go:117] "RemoveContainer" containerID="a16634b3ab0a2dd73038ba94167735afad9f7d12becc6cdf8a3fac7e653ef233"
Sep 10 21:11:13.425903 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.425887    2581 scope.go:117] "RemoveContainer" containerID="a16634b3ab0a2dd73038ba94167735afad9f7d12becc6cdf8a3fac7e653ef233"
Sep 10 21:11:13.426175 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:11:13.426159    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16634b3ab0a2dd73038ba94167735afad9f7d12becc6cdf8a3fac7e653ef233\": container with ID starting with a16634b3ab0a2dd73038ba94167735afad9f7d12becc6cdf8a3fac7e653ef233 not found: ID does not exist" containerID="a16634b3ab0a2dd73038ba94167735afad9f7d12becc6cdf8a3fac7e653ef233"
Sep 10 21:11:13.426230 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.426184    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16634b3ab0a2dd73038ba94167735afad9f7d12becc6cdf8a3fac7e653ef233"} err="failed to get container status \"a16634b3ab0a2dd73038ba94167735afad9f7d12becc6cdf8a3fac7e653ef233\": rpc error: code = NotFound desc = could not find container \"a16634b3ab0a2dd73038ba94167735afad9f7d12becc6cdf8a3fac7e653ef233\": container with ID starting with a16634b3ab0a2dd73038ba94167735afad9f7d12becc6cdf8a3fac7e653ef233 not found: ID does not exist"
Sep 10 21:11:13.437546 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.437524    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"]
Sep 10 21:11:13.442355 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.442333    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a7bbcc15-654cfc9489-p62sx"]
Sep 10 21:11:13.574275 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:13.574240    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2" path="/var/lib/kubelet/pods/a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2/volumes"
Sep 10 21:11:41.906534 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:41.906496    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"]
Sep 10 21:11:41.906926 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:41.906913    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2" containerName="maas-api"
Sep 10 21:11:41.906970 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:41.906927    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2" containerName="maas-api"
Sep 10 21:11:41.907008 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:41.906989    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a85b3a22-0325-46c4-b2f1-9f0dfc6b6cc2" containerName="maas-api"
Sep 10 21:11:41.911221 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:41.911191    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:41.913601 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:41.913583    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-869961db-kserve-self-signed-certs\""
Sep 10 21:11:41.919264 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:41.919241    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"]
Sep 10 21:11:42.048535 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.048490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52131dd3-b2cd-48f8-9793-93f91d68b300-tls-certs\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.048740 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.048541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk99l\" (UniqueName: \"kubernetes.io/projected/52131dd3-b2cd-48f8-9793-93f91d68b300-kube-api-access-nk99l\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.048740 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.048609    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-home\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.048740 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.048630    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-model-cache\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.048740 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.048690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-tmp-dir\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.048915 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.048769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-dshm\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.149888 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.149850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-tmp-dir\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.150071 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.149903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-dshm\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.150071 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.149970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52131dd3-b2cd-48f8-9793-93f91d68b300-tls-certs\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.150071 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.149998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk99l\" (UniqueName: \"kubernetes.io/projected/52131dd3-b2cd-48f8-9793-93f91d68b300-kube-api-access-nk99l\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.150071 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.150038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-home\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.150071 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.150058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-model-cache\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.150391 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.150302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-tmp-dir\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.150451 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.150403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-model-cache\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.150496 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.150453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-home\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.152234 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.152196    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-dshm\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.152370 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.152356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52131dd3-b2cd-48f8-9793-93f91d68b300-tls-certs\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.162956 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.162899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk99l\" (UniqueName: \"kubernetes.io/projected/52131dd3-b2cd-48f8-9793-93f91d68b300-kube-api-access-nk99l\") pod \"e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") " pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.221635 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.221605    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:42.492730 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.492619    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"]
Sep 10 21:11:42.495770 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:11:42.495743    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52131dd3_b2cd_48f8_9793_93f91d68b300.slice/crio-d753d16562323f03be6c7207570f4dbd4ff6f2b06c2f8fadf2d6c73cd7c016a7 WatchSource:0}: Error finding container d753d16562323f03be6c7207570f4dbd4ff6f2b06c2f8fadf2d6c73cd7c016a7: Status 404 returned error can't find the container with id d753d16562323f03be6c7207570f4dbd4ff6f2b06c2f8fadf2d6c73cd7c016a7
Sep 10 21:11:42.563522 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:42.563492    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b" event={"ID":"52131dd3-b2cd-48f8-9793-93f91d68b300","Type":"ContainerStarted","Data":"d753d16562323f03be6c7207570f4dbd4ff6f2b06c2f8fadf2d6c73cd7c016a7"}
Sep 10 21:11:43.573897 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:43.573861    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b" event={"ID":"52131dd3-b2cd-48f8-9793-93f91d68b300","Type":"ContainerStarted","Data":"d5b6efcff834eba93bcb75652416fa1ab823302a0481ba50780f0f10df3f20fb"}
Sep 10 21:11:43.587520 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:43.587473    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b" podStartSLOduration=2.587459023 podStartE2EDuration="2.587459023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:43.586127926 +0000 UTC m=+844.561787974" watchObservedRunningTime="2026-09-10 21:11:43.587459023 +0000 UTC m=+844.563119070"
Sep 10 21:11:47.809874 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:47.809846    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"]
Sep 10 21:11:47.813794 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:47.813771    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:47.816602 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:47.816573    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-63f3d7\"/\"kube-root-ca.crt\""
Sep 10 21:11:47.816730 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:47.816634    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-63f3d7\"/\"openshift-service-ca.crt\""
Sep 10 21:11:47.816884 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:47.816869    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-63f3d7\"/\"default-dockercfg-8dq2r\""
Sep 10 21:11:47.817366 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:47.817348    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-63f3d7\"/\"autores-model-63f3d7-kserve-self-signed-certs\""
Sep 10 21:11:47.825690 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:47.825392    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"]
Sep 10 21:11:47.908591 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:47.908562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t77zk\" (UniqueName: \"kubernetes.io/projected/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-kube-api-access-t77zk\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:47.908763 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:47.908600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-tmp-dir\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:47.908763 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:47.908620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-home\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:47.908763 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:47.908657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-tls-certs\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:47.908763 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:47.908742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-model-cache\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:47.908936 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:47.908823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-dshm\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:48.010280 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:48.010250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-tmp-dir\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:48.010280 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:48.010283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-home\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:48.010521 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:48.010303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-tls-certs\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:48.010521 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:48.010360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-model-cache\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:48.010521 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:48.010433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-dshm\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:48.010521 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:48.010466    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t77zk\" (UniqueName: \"kubernetes.io/projected/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-kube-api-access-t77zk\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:48.010748 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:48.010632    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-home\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:48.010748 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:48.010697    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-tmp-dir\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:48.010864 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:48.010759    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-model-cache\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:48.012570 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:48.012541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-dshm\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:48.012734 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:48.012718    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-tls-certs\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:48.018691 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:48.018669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t77zk\" (UniqueName: \"kubernetes.io/projected/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-kube-api-access-t77zk\") pod \"autores-model-63f3d7-kserve-6f5557d7ff-n4sz9\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") " pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:48.127043 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:48.126964    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:48.412185 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:48.412157    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"]
Sep 10 21:11:48.413770 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:11:48.413738    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07e6eb39_3e2a_4805_8e85_e475dfd44b0e.slice/crio-be947c9eee20a60a99c78b5552bb564a88cc73c3014eab0e48ef359945e12384 WatchSource:0}: Error finding container be947c9eee20a60a99c78b5552bb564a88cc73c3014eab0e48ef359945e12384: Status 404 returned error can't find the container with id be947c9eee20a60a99c78b5552bb564a88cc73c3014eab0e48ef359945e12384
Sep 10 21:11:48.598416 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:48.598376    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9" event={"ID":"07e6eb39-3e2a-4805-8e85-e475dfd44b0e","Type":"ContainerStarted","Data":"c2ca92fac89da0ced71e8555aae10195aa31a117b589cae71f22085ffd2dd5f7"}
Sep 10 21:11:48.598416 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:48.598421    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9" event={"ID":"07e6eb39-3e2a-4805-8e85-e475dfd44b0e","Type":"ContainerStarted","Data":"be947c9eee20a60a99c78b5552bb564a88cc73c3014eab0e48ef359945e12384"}
Sep 10 21:11:48.619062 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:48.619016    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9" podStartSLOduration=1.619001878 podStartE2EDuration="1.619001878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:48.616915596 +0000 UTC m=+849.592575637" watchObservedRunningTime="2026-09-10 21:11:48.619001878 +0000 UTC m=+849.594661925"
Sep 10 21:11:52.222538 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:52.222501    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:52.222953 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:52.222558    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:52.235328 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:52.235304    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:52.630751 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:52.630717    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:11:58.127861 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:58.127828    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:58.128288 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:58.127909    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:58.140311 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:58.140284    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:11:58.660468 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:11:58.660442    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:12:09.711376 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:09.711339    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"]
Sep 10 21:12:10.595691 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:10.595654    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"]
Sep 10 21:12:20.072155 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:20.072125    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"]
Sep 10 21:12:24.732714 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:24.732678    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b" podUID="52131dd3-b2cd-48f8-9793-93f91d68b300" containerName="main" containerID="cri-o://d5b6efcff834eba93bcb75652416fa1ab823302a0481ba50780f0f10df3f20fb" gracePeriod=45
Sep 10 21:12:24.972561 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:24.972535    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:12:25.072876 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.072846    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52131dd3-b2cd-48f8-9793-93f91d68b300-tls-certs\") pod \"52131dd3-b2cd-48f8-9793-93f91d68b300\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") "
Sep 10 21:12:25.073031 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.072897    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-model-cache\") pod \"52131dd3-b2cd-48f8-9793-93f91d68b300\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") "
Sep 10 21:12:25.073031 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.072946    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-home\") pod \"52131dd3-b2cd-48f8-9793-93f91d68b300\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") "
Sep 10 21:12:25.073031 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.073025    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-tmp-dir\") pod \"52131dd3-b2cd-48f8-9793-93f91d68b300\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") "
Sep 10 21:12:25.073224 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.073063    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk99l\" (UniqueName: \"kubernetes.io/projected/52131dd3-b2cd-48f8-9793-93f91d68b300-kube-api-access-nk99l\") pod \"52131dd3-b2cd-48f8-9793-93f91d68b300\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") "
Sep 10 21:12:25.073224 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.073087    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-dshm\") pod \"52131dd3-b2cd-48f8-9793-93f91d68b300\" (UID: \"52131dd3-b2cd-48f8-9793-93f91d68b300\") "
Sep 10 21:12:25.073346 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.073175    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-model-cache" (OuterVolumeSpecName: "model-cache") pod "52131dd3-b2cd-48f8-9793-93f91d68b300" (UID: "52131dd3-b2cd-48f8-9793-93f91d68b300"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:25.073346 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.073279    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-home" (OuterVolumeSpecName: "home") pod "52131dd3-b2cd-48f8-9793-93f91d68b300" (UID: "52131dd3-b2cd-48f8-9793-93f91d68b300"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:25.073346 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.073309    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52131dd3-b2cd-48f8-9793-93f91d68b300" (UID: "52131dd3-b2cd-48f8-9793-93f91d68b300"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:25.073533 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.073511    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-tmp-dir\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:25.073582 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.073538    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-model-cache\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:25.073582 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.073552    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-home\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:25.075134 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.075106    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52131dd3-b2cd-48f8-9793-93f91d68b300-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52131dd3-b2cd-48f8-9793-93f91d68b300" (UID: "52131dd3-b2cd-48f8-9793-93f91d68b300"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:12:25.075134 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.075130    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-dshm" (OuterVolumeSpecName: "dshm") pod "52131dd3-b2cd-48f8-9793-93f91d68b300" (UID: "52131dd3-b2cd-48f8-9793-93f91d68b300"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:25.075297 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.075217    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52131dd3-b2cd-48f8-9793-93f91d68b300-kube-api-access-nk99l" (OuterVolumeSpecName: "kube-api-access-nk99l") pod "52131dd3-b2cd-48f8-9793-93f91d68b300" (UID: "52131dd3-b2cd-48f8-9793-93f91d68b300"). InnerVolumeSpecName "kube-api-access-nk99l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:12:25.174667 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.174634    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk99l\" (UniqueName: \"kubernetes.io/projected/52131dd3-b2cd-48f8-9793-93f91d68b300-kube-api-access-nk99l\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:25.174667 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.174662    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52131dd3-b2cd-48f8-9793-93f91d68b300-dshm\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:25.174859 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.174676    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52131dd3-b2cd-48f8-9793-93f91d68b300-tls-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:25.614872 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.614810    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9" podUID="07e6eb39-3e2a-4805-8e85-e475dfd44b0e" containerName="main" containerID="cri-o://c2ca92fac89da0ced71e8555aae10195aa31a117b589cae71f22085ffd2dd5f7" gracePeriod=45
Sep 10 21:12:25.787058 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.787031    2581 generic.go:356] "Generic (PLEG): container finished" podID="07e6eb39-3e2a-4805-8e85-e475dfd44b0e" containerID="c2ca92fac89da0ced71e8555aae10195aa31a117b589cae71f22085ffd2dd5f7" exitCode=0
Sep 10 21:12:25.787433 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.787118    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9" event={"ID":"07e6eb39-3e2a-4805-8e85-e475dfd44b0e","Type":"ContainerDied","Data":"c2ca92fac89da0ced71e8555aae10195aa31a117b589cae71f22085ffd2dd5f7"}
Sep 10 21:12:25.788551 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.788530    2581 generic.go:356] "Generic (PLEG): container finished" podID="52131dd3-b2cd-48f8-9793-93f91d68b300" containerID="d5b6efcff834eba93bcb75652416fa1ab823302a0481ba50780f0f10df3f20fb" exitCode=0
Sep 10 21:12:25.788666 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.788559    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b" event={"ID":"52131dd3-b2cd-48f8-9793-93f91d68b300","Type":"ContainerDied","Data":"d5b6efcff834eba93bcb75652416fa1ab823302a0481ba50780f0f10df3f20fb"}
Sep 10 21:12:25.788666 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.788576    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"
Sep 10 21:12:25.788666 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.788612    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b" event={"ID":"52131dd3-b2cd-48f8-9793-93f91d68b300","Type":"ContainerDied","Data":"d753d16562323f03be6c7207570f4dbd4ff6f2b06c2f8fadf2d6c73cd7c016a7"}
Sep 10 21:12:25.788666 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.788636    2581 scope.go:117] "RemoveContainer" containerID="d5b6efcff834eba93bcb75652416fa1ab823302a0481ba50780f0f10df3f20fb"
Sep 10 21:12:25.801024 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.800976    2581 scope.go:117] "RemoveContainer" containerID="d5b6efcff834eba93bcb75652416fa1ab823302a0481ba50780f0f10df3f20fb"
Sep 10 21:12:25.801400 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:12:25.801379    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b6efcff834eba93bcb75652416fa1ab823302a0481ba50780f0f10df3f20fb\": container with ID starting with d5b6efcff834eba93bcb75652416fa1ab823302a0481ba50780f0f10df3f20fb not found: ID does not exist" containerID="d5b6efcff834eba93bcb75652416fa1ab823302a0481ba50780f0f10df3f20fb"
Sep 10 21:12:25.801496 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.801410    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b6efcff834eba93bcb75652416fa1ab823302a0481ba50780f0f10df3f20fb"} err="failed to get container status \"d5b6efcff834eba93bcb75652416fa1ab823302a0481ba50780f0f10df3f20fb\": rpc error: code = NotFound desc = could not find container \"d5b6efcff834eba93bcb75652416fa1ab823302a0481ba50780f0f10df3f20fb\": container with ID starting with d5b6efcff834eba93bcb75652416fa1ab823302a0481ba50780f0f10df3f20fb not found: ID does not exist"
Sep 10 21:12:25.806163 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.806142    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"]
Sep 10 21:12:25.808069 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.808044    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-869961db-kserve-7989975cbb-6wl8b"]
Sep 10 21:12:25.872638 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.872614    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:12:25.982884 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.982856    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-dshm\") pod \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") "
Sep 10 21:12:25.983048 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.982895    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t77zk\" (UniqueName: \"kubernetes.io/projected/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-kube-api-access-t77zk\") pod \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") "
Sep 10 21:12:25.983048 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.982935    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-home\") pod \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") "
Sep 10 21:12:25.983048 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.982988    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-model-cache\") pod \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") "
Sep 10 21:12:25.983048 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.983022    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-tls-certs\") pod \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") "
Sep 10 21:12:25.983350 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.983079    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-tmp-dir\") pod \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\" (UID: \"07e6eb39-3e2a-4805-8e85-e475dfd44b0e\") "
Sep 10 21:12:25.983350 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.983253    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-home" (OuterVolumeSpecName: "home") pod "07e6eb39-3e2a-4805-8e85-e475dfd44b0e" (UID: "07e6eb39-3e2a-4805-8e85-e475dfd44b0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:25.983350 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.983278    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-model-cache" (OuterVolumeSpecName: "model-cache") pod "07e6eb39-3e2a-4805-8e85-e475dfd44b0e" (UID: "07e6eb39-3e2a-4805-8e85-e475dfd44b0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:25.983497 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.983403    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07e6eb39-3e2a-4805-8e85-e475dfd44b0e" (UID: "07e6eb39-3e2a-4805-8e85-e475dfd44b0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:25.983497 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.983472    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-home\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:25.983497 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.983486    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-model-cache\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:25.985282 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.985248    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-kube-api-access-t77zk" (OuterVolumeSpecName: "kube-api-access-t77zk") pod "07e6eb39-3e2a-4805-8e85-e475dfd44b0e" (UID: "07e6eb39-3e2a-4805-8e85-e475dfd44b0e"). InnerVolumeSpecName "kube-api-access-t77zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:12:25.985411 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.985394    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-dshm" (OuterVolumeSpecName: "dshm") pod "07e6eb39-3e2a-4805-8e85-e475dfd44b0e" (UID: "07e6eb39-3e2a-4805-8e85-e475dfd44b0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:25.985535 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:25.985520    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07e6eb39-3e2a-4805-8e85-e475dfd44b0e" (UID: "07e6eb39-3e2a-4805-8e85-e475dfd44b0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:12:26.084747 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:26.084720    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-dshm\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:26.084747 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:26.084744    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t77zk\" (UniqueName: \"kubernetes.io/projected/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-kube-api-access-t77zk\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:26.084935 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:26.084755    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-tls-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:26.084935 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:26.084763    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07e6eb39-3e2a-4805-8e85-e475dfd44b0e-tmp-dir\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:26.796013 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:26.795987    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"
Sep 10 21:12:26.796481 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:26.796033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9" event={"ID":"07e6eb39-3e2a-4805-8e85-e475dfd44b0e","Type":"ContainerDied","Data":"be947c9eee20a60a99c78b5552bb564a88cc73c3014eab0e48ef359945e12384"}
Sep 10 21:12:26.796481 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:26.796093    2581 scope.go:117] "RemoveContainer" containerID="c2ca92fac89da0ced71e8555aae10195aa31a117b589cae71f22085ffd2dd5f7"
Sep 10 21:12:26.817297 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:26.817273    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"]
Sep 10 21:12:26.822192 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:26.822173    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-63f3d7/autores-model-63f3d7-kserve-6f5557d7ff-n4sz9"]
Sep 10 21:12:27.574184 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:27.574134    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07e6eb39-3e2a-4805-8e85-e475dfd44b0e" path="/var/lib/kubelet/pods/07e6eb39-3e2a-4805-8e85-e475dfd44b0e/volumes"
Sep 10 21:12:27.574685 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:27.574663    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52131dd3-b2cd-48f8-9793-93f91d68b300" path="/var/lib/kubelet/pods/52131dd3-b2cd-48f8-9793-93f91d68b300/volumes"
Sep 10 21:12:35.091323 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.091219    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl" podUID="3488c548-8089-4422-a20d-67d8505f2b82" containerName="main" containerID="cri-o://79ad9354ba383ef0dfe3909922cf68326a2d6f64326cc9566f1028a50d2bd660" gracePeriod=45
Sep 10 21:12:35.342725 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.342661    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:12:35.370510 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.370483    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs8x5\" (UniqueName: \"kubernetes.io/projected/3488c548-8089-4422-a20d-67d8505f2b82-kube-api-access-bs8x5\") pod \"3488c548-8089-4422-a20d-67d8505f2b82\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") "
Sep 10 21:12:35.370676 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.370568    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-home\") pod \"3488c548-8089-4422-a20d-67d8505f2b82\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") "
Sep 10 21:12:35.370676 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.370606    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-model-cache\") pod \"3488c548-8089-4422-a20d-67d8505f2b82\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") "
Sep 10 21:12:35.370676 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.370637    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3488c548-8089-4422-a20d-67d8505f2b82-tls-certs\") pod \"3488c548-8089-4422-a20d-67d8505f2b82\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") "
Sep 10 21:12:35.370851 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.370678    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-dshm\") pod \"3488c548-8089-4422-a20d-67d8505f2b82\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") "
Sep 10 21:12:35.370851 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.370719    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-tmp-dir\") pod \"3488c548-8089-4422-a20d-67d8505f2b82\" (UID: \"3488c548-8089-4422-a20d-67d8505f2b82\") "
Sep 10 21:12:35.370960 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.370857    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-home" (OuterVolumeSpecName: "home") pod "3488c548-8089-4422-a20d-67d8505f2b82" (UID: "3488c548-8089-4422-a20d-67d8505f2b82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:35.370960 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.370867    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-model-cache" (OuterVolumeSpecName: "model-cache") pod "3488c548-8089-4422-a20d-67d8505f2b82" (UID: "3488c548-8089-4422-a20d-67d8505f2b82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:35.371060 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.371013    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-home\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:35.371060 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.371031    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-model-cache\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:35.371060 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.371040    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3488c548-8089-4422-a20d-67d8505f2b82" (UID: "3488c548-8089-4422-a20d-67d8505f2b82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:35.372796 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.372765    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-dshm" (OuterVolumeSpecName: "dshm") pod "3488c548-8089-4422-a20d-67d8505f2b82" (UID: "3488c548-8089-4422-a20d-67d8505f2b82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:12:35.372913 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.372833    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3488c548-8089-4422-a20d-67d8505f2b82-kube-api-access-bs8x5" (OuterVolumeSpecName: "kube-api-access-bs8x5") pod "3488c548-8089-4422-a20d-67d8505f2b82" (UID: "3488c548-8089-4422-a20d-67d8505f2b82"). InnerVolumeSpecName "kube-api-access-bs8x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:12:35.372913 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.372879    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3488c548-8089-4422-a20d-67d8505f2b82-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3488c548-8089-4422-a20d-67d8505f2b82" (UID: "3488c548-8089-4422-a20d-67d8505f2b82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:12:35.472214 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.472179    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-dshm\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:35.472355 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.472230    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3488c548-8089-4422-a20d-67d8505f2b82-tmp-dir\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:35.472355 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.472240    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs8x5\" (UniqueName: \"kubernetes.io/projected/3488c548-8089-4422-a20d-67d8505f2b82-kube-api-access-bs8x5\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:35.472355 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.472250    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3488c548-8089-4422-a20d-67d8505f2b82-tls-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:12:35.843393 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.843359    2581 generic.go:356] "Generic (PLEG): container finished" podID="3488c548-8089-4422-a20d-67d8505f2b82" containerID="79ad9354ba383ef0dfe3909922cf68326a2d6f64326cc9566f1028a50d2bd660" exitCode=0
Sep 10 21:12:35.843549 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.843400    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"
Sep 10 21:12:35.843549 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.843444    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl" event={"ID":"3488c548-8089-4422-a20d-67d8505f2b82","Type":"ContainerDied","Data":"79ad9354ba383ef0dfe3909922cf68326a2d6f64326cc9566f1028a50d2bd660"}
Sep 10 21:12:35.843549 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.843485    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl" event={"ID":"3488c548-8089-4422-a20d-67d8505f2b82","Type":"ContainerDied","Data":"8e637fb5f4b7c0437834b5b0ccf53fa8a2fb95cd557661c9cbf37754bc874c84"}
Sep 10 21:12:35.843549 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.843502    2581 scope.go:117] "RemoveContainer" containerID="79ad9354ba383ef0dfe3909922cf68326a2d6f64326cc9566f1028a50d2bd660"
Sep 10 21:12:35.851559 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.851542    2581 scope.go:117] "RemoveContainer" containerID="79ad9354ba383ef0dfe3909922cf68326a2d6f64326cc9566f1028a50d2bd660"
Sep 10 21:12:35.851805 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:12:35.851788    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ad9354ba383ef0dfe3909922cf68326a2d6f64326cc9566f1028a50d2bd660\": container with ID starting with 79ad9354ba383ef0dfe3909922cf68326a2d6f64326cc9566f1028a50d2bd660 not found: ID does not exist" containerID="79ad9354ba383ef0dfe3909922cf68326a2d6f64326cc9566f1028a50d2bd660"
Sep 10 21:12:35.851855 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.851813    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ad9354ba383ef0dfe3909922cf68326a2d6f64326cc9566f1028a50d2bd660"} err="failed to get container status \"79ad9354ba383ef0dfe3909922cf68326a2d6f64326cc9566f1028a50d2bd660\": rpc error: code = NotFound desc = could not find container \"79ad9354ba383ef0dfe3909922cf68326a2d6f64326cc9566f1028a50d2bd660\": container with ID starting with 79ad9354ba383ef0dfe3909922cf68326a2d6f64326cc9566f1028a50d2bd660 not found: ID does not exist"
Sep 10 21:12:35.865039 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.865015    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"]
Sep 10 21:12:35.871082 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:35.871052    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-869961db-kserve-6997798744-f6kwl"]
Sep 10 21:12:36.266029 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.265955    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"]
Sep 10 21:12:36.266410 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.266351    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52131dd3-b2cd-48f8-9793-93f91d68b300" containerName="main"
Sep 10 21:12:36.266410 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.266363    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="52131dd3-b2cd-48f8-9793-93f91d68b300" containerName="main"
Sep 10 21:12:36.266410 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.266381    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3488c548-8089-4422-a20d-67d8505f2b82" containerName="main"
Sep 10 21:12:36.266410 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.266386    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3488c548-8089-4422-a20d-67d8505f2b82" containerName="main"
Sep 10 21:12:36.266410 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.266396    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e6eb39-3e2a-4805-8e85-e475dfd44b0e" containerName="main"
Sep 10 21:12:36.266410 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.266401    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e6eb39-3e2a-4805-8e85-e475dfd44b0e" containerName="main"
Sep 10 21:12:36.266601 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.266457    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3488c548-8089-4422-a20d-67d8505f2b82" containerName="main"
Sep 10 21:12:36.266601 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.266467    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="52131dd3-b2cd-48f8-9793-93f91d68b300" containerName="main"
Sep 10 21:12:36.266601 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.266474    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e6eb39-3e2a-4805-8e85-e475dfd44b0e" containerName="main"
Sep 10 21:12:36.269422 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.269406    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"
Sep 10 21:12:36.271608 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.271581    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-cb845d\""
Sep 10 21:12:36.271746 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.271659    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-cb845d-dockercfg-shk7s\""
Sep 10 21:12:36.280463 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.280431    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"]
Sep 10 21:12:36.381258 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.381224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe4162a-4cca-4712-bcbb-76ea29804448-plugins-config-volume\") pod \"payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8\" (UID: \"afe4162a-4cca-4712-bcbb-76ea29804448\") " pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"
Sep 10 21:12:36.381446 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.381297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4qnz\" (UniqueName: \"kubernetes.io/projected/afe4162a-4cca-4712-bcbb-76ea29804448-kube-api-access-m4qnz\") pod \"payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8\" (UID: \"afe4162a-4cca-4712-bcbb-76ea29804448\") " pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"
Sep 10 21:12:36.482686 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.482651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe4162a-4cca-4712-bcbb-76ea29804448-plugins-config-volume\") pod \"payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8\" (UID: \"afe4162a-4cca-4712-bcbb-76ea29804448\") " pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"
Sep 10 21:12:36.482857 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.482715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4qnz\" (UniqueName: \"kubernetes.io/projected/afe4162a-4cca-4712-bcbb-76ea29804448-kube-api-access-m4qnz\") pod \"payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8\" (UID: \"afe4162a-4cca-4712-bcbb-76ea29804448\") " pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"
Sep 10 21:12:36.483342 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.483320    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe4162a-4cca-4712-bcbb-76ea29804448-plugins-config-volume\") pod \"payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8\" (UID: \"afe4162a-4cca-4712-bcbb-76ea29804448\") " pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"
Sep 10 21:12:36.490953 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.490928    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4qnz\" (UniqueName: \"kubernetes.io/projected/afe4162a-4cca-4712-bcbb-76ea29804448-kube-api-access-m4qnz\") pod \"payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8\" (UID: \"afe4162a-4cca-4712-bcbb-76ea29804448\") " pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"
Sep 10 21:12:36.580509 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.580483    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"
Sep 10 21:12:36.857655 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:36.857584    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"]
Sep 10 21:12:36.860461 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:12:36.860434    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe4162a_4cca_4712_bcbb_76ea29804448.slice/crio-8a18f25c899cecb4736ad15285b7f6c0a16dcb758a1e3ee9b53ec9c13bfbedd2 WatchSource:0}: Error finding container 8a18f25c899cecb4736ad15285b7f6c0a16dcb758a1e3ee9b53ec9c13bfbedd2: Status 404 returned error can't find the container with id 8a18f25c899cecb4736ad15285b7f6c0a16dcb758a1e3ee9b53ec9c13bfbedd2
Sep 10 21:12:37.574217 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:37.574170    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3488c548-8089-4422-a20d-67d8505f2b82" path="/var/lib/kubelet/pods/3488c548-8089-4422-a20d-67d8505f2b82/volumes"
Sep 10 21:12:37.855273 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:37.855169    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8" event={"ID":"afe4162a-4cca-4712-bcbb-76ea29804448","Type":"ContainerStarted","Data":"d88c83fc68e3b697f63e60ae491485b93d781627b7f237af02d35092e63b828e"}
Sep 10 21:12:37.855273 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:37.855249    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"
Sep 10 21:12:37.855437 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:37.855278    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8" event={"ID":"afe4162a-4cca-4712-bcbb-76ea29804448","Type":"ContainerStarted","Data":"8a18f25c899cecb4736ad15285b7f6c0a16dcb758a1e3ee9b53ec9c13bfbedd2"}
Sep 10 21:12:37.878533 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:37.878487    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8" podStartSLOduration=1.8784728149999999 podStartE2EDuration="1.878472815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:37.877762436 +0000 UTC m=+898.853422483" watchObservedRunningTime="2026-09-10 21:12:37.878472815 +0000 UTC m=+898.854132859"
Sep 10 21:12:48.863158 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:12:48.863127    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"
Sep 10 21:13:06.005388 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.005356    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"]
Sep 10 21:13:06.008678 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.008660    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.011713 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.011689    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-cb845d\"/\"default-dockercfg-c7qr2\""
Sep 10 21:13:06.011875 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.011722    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-cb845d\"/\"explicit-model-cb845d-kserve-self-signed-certs\""
Sep 10 21:13:06.011875 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.011757    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-cb845d\"/\"kube-root-ca.crt\""
Sep 10 21:13:06.011875 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.011689    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-cb845d\"/\"openshift-service-ca.crt\""
Sep 10 21:13:06.018712 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.018690    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"]
Sep 10 21:13:06.055876 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.055843    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmz2d\" (UniqueName: \"kubernetes.io/projected/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-kube-api-access-mmz2d\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.055876 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.055875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-tls-certs\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.056108 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.055908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-model-cache\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.056108 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.055924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-tmp-dir\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.056108 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.056026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-dshm\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.056108 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.056059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-home\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.157013 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.156980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-dshm\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.157013 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.157012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-home\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.157289 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.157078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmz2d\" (UniqueName: \"kubernetes.io/projected/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-kube-api-access-mmz2d\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.157289 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.157097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-tls-certs\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.157406 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.157298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-model-cache\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.157406 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.157318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-tmp-dir\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.157523 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.157498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-home\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.157680 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.157652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-tmp-dir\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.157680 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.157673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-model-cache\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.159298 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.159276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-dshm\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.159472 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.159454    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-tls-certs\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.165684 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.165667    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmz2d\" (UniqueName: \"kubernetes.io/projected/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-kube-api-access-mmz2d\") pod \"explicit-model-cb845d-kserve-5bb86444f-czjmg\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") " pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.319613 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.319573    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:06.592757 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:06.592735    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"]
Sep 10 21:13:06.594509 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:13:06.594479    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3fc2cd4_c803_414e_9ed1_c19f3fc97100.slice/crio-8a8fbfdf6ddc4d0ecac96703c70ff622318fb01f623a2d1b6ce4a9a89575c062 WatchSource:0}: Error finding container 8a8fbfdf6ddc4d0ecac96703c70ff622318fb01f623a2d1b6ce4a9a89575c062: Status 404 returned error can't find the container with id 8a8fbfdf6ddc4d0ecac96703c70ff622318fb01f623a2d1b6ce4a9a89575c062
Sep 10 21:13:07.003719 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:07.003635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg" event={"ID":"c3fc2cd4-c803-414e-9ed1-c19f3fc97100","Type":"ContainerStarted","Data":"1df280d20da63e729549d8949a83b85584ee34d1f68a02e5ca5a9cf1ecdbf056"}
Sep 10 21:13:07.003719 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:07.003679    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg" event={"ID":"c3fc2cd4-c803-414e-9ed1-c19f3fc97100","Type":"ContainerStarted","Data":"8a8fbfdf6ddc4d0ecac96703c70ff622318fb01f623a2d1b6ce4a9a89575c062"}
Sep 10 21:13:07.022148 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:07.022053    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg" podStartSLOduration=2.022036829 podStartE2EDuration="2.022036829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:07.021581443 +0000 UTC m=+927.997241501" watchObservedRunningTime="2026-09-10 21:13:07.022036829 +0000 UTC m=+927.997696876"
Sep 10 21:13:16.320411 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:16.320376    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:16.320867 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:16.320424    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:16.332880 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:16.332855    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:17.065961 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:17.065931    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:19.250252 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:19.250213    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"]
Sep 10 21:13:29.934088 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:29.934052    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"]
Sep 10 21:13:29.934552 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:29.934310    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8" podUID="afe4162a-4cca-4712-bcbb-76ea29804448" containerName="payload-processing" containerID="cri-o://d88c83fc68e3b697f63e60ae491485b93d781627b7f237af02d35092e63b828e" gracePeriod=30
Sep 10 21:13:30.117431 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:30.117402    2581 generic.go:356] "Generic (PLEG): container finished" podID="afe4162a-4cca-4712-bcbb-76ea29804448" containerID="d88c83fc68e3b697f63e60ae491485b93d781627b7f237af02d35092e63b828e" exitCode=0
Sep 10 21:13:30.117546 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:30.117446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8" event={"ID":"afe4162a-4cca-4712-bcbb-76ea29804448","Type":"ContainerDied","Data":"d88c83fc68e3b697f63e60ae491485b93d781627b7f237af02d35092e63b828e"}
Sep 10 21:13:30.180106 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:30.180077    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"
Sep 10 21:13:30.271231 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:30.271133    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe4162a-4cca-4712-bcbb-76ea29804448-plugins-config-volume\") pod \"afe4162a-4cca-4712-bcbb-76ea29804448\" (UID: \"afe4162a-4cca-4712-bcbb-76ea29804448\") "
Sep 10 21:13:30.271231 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:30.271195    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4qnz\" (UniqueName: \"kubernetes.io/projected/afe4162a-4cca-4712-bcbb-76ea29804448-kube-api-access-m4qnz\") pod \"afe4162a-4cca-4712-bcbb-76ea29804448\" (UID: \"afe4162a-4cca-4712-bcbb-76ea29804448\") "
Sep 10 21:13:30.271525 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:30.271502    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe4162a-4cca-4712-bcbb-76ea29804448-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "afe4162a-4cca-4712-bcbb-76ea29804448" (UID: "afe4162a-4cca-4712-bcbb-76ea29804448"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:30.273313 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:30.273291    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe4162a-4cca-4712-bcbb-76ea29804448-kube-api-access-m4qnz" (OuterVolumeSpecName: "kube-api-access-m4qnz") pod "afe4162a-4cca-4712-bcbb-76ea29804448" (UID: "afe4162a-4cca-4712-bcbb-76ea29804448"). InnerVolumeSpecName "kube-api-access-m4qnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:30.372308 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:30.372282    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe4162a-4cca-4712-bcbb-76ea29804448-plugins-config-volume\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:13:30.372308 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:30.372306    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4qnz\" (UniqueName: \"kubernetes.io/projected/afe4162a-4cca-4712-bcbb-76ea29804448-kube-api-access-m4qnz\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:13:31.124026 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:31.123998    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"
Sep 10 21:13:31.124026 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:31.124012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8" event={"ID":"afe4162a-4cca-4712-bcbb-76ea29804448","Type":"ContainerDied","Data":"8a18f25c899cecb4736ad15285b7f6c0a16dcb758a1e3ee9b53ec9c13bfbedd2"}
Sep 10 21:13:31.124553 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:31.124065    2581 scope.go:117] "RemoveContainer" containerID="d88c83fc68e3b697f63e60ae491485b93d781627b7f237af02d35092e63b828e"
Sep 10 21:13:31.152535 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:31.152506    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"]
Sep 10 21:13:31.158499 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:31.158477    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cb845d-8b6c47cb9-gvgx8"]
Sep 10 21:13:31.575008 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:31.574967    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afe4162a-4cca-4712-bcbb-76ea29804448" path="/var/lib/kubelet/pods/afe4162a-4cca-4712-bcbb-76ea29804448/volumes"
Sep 10 21:13:34.269237 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.269170    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg" podUID="c3fc2cd4-c803-414e-9ed1-c19f3fc97100" containerName="main" containerID="cri-o://1df280d20da63e729549d8949a83b85584ee34d1f68a02e5ca5a9cf1ecdbf056" gracePeriod=45
Sep 10 21:13:34.526710 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.526652    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:34.612390 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.612359    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-dshm\") pod \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") "
Sep 10 21:13:34.612553 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.612421    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-tmp-dir\") pod \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") "
Sep 10 21:13:34.612553 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.612456    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-home\") pod \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") "
Sep 10 21:13:34.612553 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.612484    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-tls-certs\") pod \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") "
Sep 10 21:13:34.612553 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.612549    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-model-cache\") pod \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") "
Sep 10 21:13:34.612756 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.612613    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmz2d\" (UniqueName: \"kubernetes.io/projected/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-kube-api-access-mmz2d\") pod \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\" (UID: \"c3fc2cd4-c803-414e-9ed1-c19f3fc97100\") "
Sep 10 21:13:34.612756 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.612702    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3fc2cd4-c803-414e-9ed1-c19f3fc97100" (UID: "c3fc2cd4-c803-414e-9ed1-c19f3fc97100"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:34.612756 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.612714    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-home" (OuterVolumeSpecName: "home") pod "c3fc2cd4-c803-414e-9ed1-c19f3fc97100" (UID: "c3fc2cd4-c803-414e-9ed1-c19f3fc97100"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:34.612910 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.612836    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3fc2cd4-c803-414e-9ed1-c19f3fc97100" (UID: "c3fc2cd4-c803-414e-9ed1-c19f3fc97100"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:34.612973 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.612934    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-tmp-dir\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:13:34.612973 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.612948    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-home\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:13:34.612973 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.612965    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-model-cache\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:13:34.614722 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.614696    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3fc2cd4-c803-414e-9ed1-c19f3fc97100" (UID: "c3fc2cd4-c803-414e-9ed1-c19f3fc97100"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:13:34.614832 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.614729    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-kube-api-access-mmz2d" (OuterVolumeSpecName: "kube-api-access-mmz2d") pod "c3fc2cd4-c803-414e-9ed1-c19f3fc97100" (UID: "c3fc2cd4-c803-414e-9ed1-c19f3fc97100"). InnerVolumeSpecName "kube-api-access-mmz2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:34.614832 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.614728    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-dshm" (OuterVolumeSpecName: "dshm") pod "c3fc2cd4-c803-414e-9ed1-c19f3fc97100" (UID: "c3fc2cd4-c803-414e-9ed1-c19f3fc97100"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:34.713562 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.713531    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-dshm\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:13:34.713562 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.713556    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-tls-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:13:34.713562 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:34.713567    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmz2d\" (UniqueName: \"kubernetes.io/projected/c3fc2cd4-c803-414e-9ed1-c19f3fc97100-kube-api-access-mmz2d\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:13:35.146893 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:35.146856    2581 generic.go:356] "Generic (PLEG): container finished" podID="c3fc2cd4-c803-414e-9ed1-c19f3fc97100" containerID="1df280d20da63e729549d8949a83b85584ee34d1f68a02e5ca5a9cf1ecdbf056" exitCode=0
Sep 10 21:13:35.146893 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:35.146895    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"
Sep 10 21:13:35.147126 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:35.146939    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg" event={"ID":"c3fc2cd4-c803-414e-9ed1-c19f3fc97100","Type":"ContainerDied","Data":"1df280d20da63e729549d8949a83b85584ee34d1f68a02e5ca5a9cf1ecdbf056"}
Sep 10 21:13:35.147126 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:35.146981    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg" event={"ID":"c3fc2cd4-c803-414e-9ed1-c19f3fc97100","Type":"ContainerDied","Data":"8a8fbfdf6ddc4d0ecac96703c70ff622318fb01f623a2d1b6ce4a9a89575c062"}
Sep 10 21:13:35.147126 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:35.146996    2581 scope.go:117] "RemoveContainer" containerID="1df280d20da63e729549d8949a83b85584ee34d1f68a02e5ca5a9cf1ecdbf056"
Sep 10 21:13:35.155956 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:35.155940    2581 scope.go:117] "RemoveContainer" containerID="1df280d20da63e729549d8949a83b85584ee34d1f68a02e5ca5a9cf1ecdbf056"
Sep 10 21:13:35.156217 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:13:35.156180    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df280d20da63e729549d8949a83b85584ee34d1f68a02e5ca5a9cf1ecdbf056\": container with ID starting with 1df280d20da63e729549d8949a83b85584ee34d1f68a02e5ca5a9cf1ecdbf056 not found: ID does not exist" containerID="1df280d20da63e729549d8949a83b85584ee34d1f68a02e5ca5a9cf1ecdbf056"
Sep 10 21:13:35.156265 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:35.156224    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df280d20da63e729549d8949a83b85584ee34d1f68a02e5ca5a9cf1ecdbf056"} err="failed to get container status \"1df280d20da63e729549d8949a83b85584ee34d1f68a02e5ca5a9cf1ecdbf056\": rpc error: code = NotFound desc = could not find container \"1df280d20da63e729549d8949a83b85584ee34d1f68a02e5ca5a9cf1ecdbf056\": container with ID starting with 1df280d20da63e729549d8949a83b85584ee34d1f68a02e5ca5a9cf1ecdbf056 not found: ID does not exist"
Sep 10 21:13:35.170980 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:35.170960    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"]
Sep 10 21:13:35.180681 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:35.180661    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-cb845d/explicit-model-cb845d-kserve-5bb86444f-czjmg"]
Sep 10 21:13:35.574107 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:35.574078    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3fc2cd4-c803-414e-9ed1-c19f3fc97100" path="/var/lib/kubelet/pods/c3fc2cd4-c803-414e-9ed1-c19f3fc97100/volumes"
Sep 10 21:13:43.613773 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.613738    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"]
Sep 10 21:13:43.614134 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.614126    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afe4162a-4cca-4712-bcbb-76ea29804448" containerName="payload-processing"
Sep 10 21:13:43.614180 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.614136    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="afe4162a-4cca-4712-bcbb-76ea29804448" containerName="payload-processing"
Sep 10 21:13:43.614180 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.614157    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3fc2cd4-c803-414e-9ed1-c19f3fc97100" containerName="main"
Sep 10 21:13:43.614180 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.614163    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3fc2cd4-c803-414e-9ed1-c19f3fc97100" containerName="main"
Sep 10 21:13:43.614310 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.614240    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="afe4162a-4cca-4712-bcbb-76ea29804448" containerName="payload-processing"
Sep 10 21:13:43.614310 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.614250    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3fc2cd4-c803-414e-9ed1-c19f3fc97100" containerName="main"
Sep 10 21:13:43.618578 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.618557    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.625603 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.622960    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-7cc2bd\"/\"default-dockercfg-t6h82\""
Sep 10 21:13:43.625603 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.623242    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-7cc2bd\"/\"noten-model-7cc2bd-kserve-self-signed-certs\""
Sep 10 21:13:43.625603 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.623295    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-7cc2bd\"/\"kube-root-ca.crt\""
Sep 10 21:13:43.625603 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.623571    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-7cc2bd\"/\"openshift-service-ca.crt\""
Sep 10 21:13:43.629289 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.629269    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"]
Sep 10 21:13:43.801175 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.801138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-tls-certs\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.801372 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.801186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-home\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.801372 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.801232    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-model-cache\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.801372 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.801260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5msb\" (UniqueName: \"kubernetes.io/projected/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-kube-api-access-m5msb\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.801372 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.801287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-tmp-dir\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.801372 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.801306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-dshm\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.902225 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.902128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-model-cache\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.902225 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.902170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5msb\" (UniqueName: \"kubernetes.io/projected/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-kube-api-access-m5msb\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.902419 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.902363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-tmp-dir\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.902419 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.902390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-dshm\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.902491 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.902475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-tls-certs\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.902531 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.902501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-model-cache\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.902531 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.902527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-home\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.902740 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.902723    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-tmp-dir\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.902836 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.902822    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-home\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.904578 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.904548    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-dshm\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.904802 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.904785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-tls-certs\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.910454 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.910434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5msb\" (UniqueName: \"kubernetes.io/projected/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-kube-api-access-m5msb\") pod \"noten-model-7cc2bd-kserve-94b56745f-qmz6c\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") " pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:43.930957 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:43.930935    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:13:44.203307 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:44.203233    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"]
Sep 10 21:13:44.206673 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:13:44.206648    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffdb74ac_10d0_4ab7_b0a5_eee66b1f2925.slice/crio-7cc840aa834919f2be24e97e708bf8efe8a1fb4fe0d93e6d3b897b4539478899 WatchSource:0}: Error finding container 7cc840aa834919f2be24e97e708bf8efe8a1fb4fe0d93e6d3b897b4539478899: Status 404 returned error can't find the container with id 7cc840aa834919f2be24e97e708bf8efe8a1fb4fe0d93e6d3b897b4539478899
Sep 10 21:13:45.196022 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:45.195989    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c" event={"ID":"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925","Type":"ContainerStarted","Data":"590d9a6b53ba874ab4bbf30000ababca945f0b762148ed79c846fbf3654c4fe1"}
Sep 10 21:13:45.196022 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:45.196028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c" event={"ID":"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925","Type":"ContainerStarted","Data":"7cc840aa834919f2be24e97e708bf8efe8a1fb4fe0d93e6d3b897b4539478899"}
Sep 10 21:13:45.221667 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:45.221622    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c" podStartSLOduration=2.221609719 podStartE2EDuration="2.221609719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:45.218457065 +0000 UTC m=+966.194117111" watchObservedRunningTime="2026-09-10 21:13:45.221609719 +0000 UTC m=+966.197269765"
Sep 10 21:13:51.675128 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:51.675099    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"]
Sep 10 21:13:53.931522 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:13:53.931492    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:14:03.342739 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:03.342650    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"]
Sep 10 21:14:03.343120 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:03.342846    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf" podUID="93023191-175a-4709-9855-e154821a741e" containerName="maas-api" containerID="cri-o://a997db38d9a89b915c69ee7baf0f53f4fc5f062f7b4ce2cc27b129dcc11b3b31" gracePeriod=30
Sep 10 21:14:03.590099 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:03.590075    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:14:03.680702 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:03.680600    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93023191-175a-4709-9855-e154821a741e-metrics-certs\") pod \"93023191-175a-4709-9855-e154821a741e\" (UID: \"93023191-175a-4709-9855-e154821a741e\") "
Sep 10 21:14:03.680702 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:03.680708    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93023191-175a-4709-9855-e154821a741e-maas-api-tls\") pod \"93023191-175a-4709-9855-e154821a741e\" (UID: \"93023191-175a-4709-9855-e154821a741e\") "
Sep 10 21:14:03.680897 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:03.680756    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93023191-175a-4709-9855-e154821a741e-openshift-service-ca\") pod \"93023191-175a-4709-9855-e154821a741e\" (UID: \"93023191-175a-4709-9855-e154821a741e\") "
Sep 10 21:14:03.680897 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:03.680789    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn7lq\" (UniqueName: \"kubernetes.io/projected/93023191-175a-4709-9855-e154821a741e-kube-api-access-zn7lq\") pod \"93023191-175a-4709-9855-e154821a741e\" (UID: \"93023191-175a-4709-9855-e154821a741e\") "
Sep 10 21:14:03.681147 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:03.681124    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93023191-175a-4709-9855-e154821a741e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "93023191-175a-4709-9855-e154821a741e" (UID: "93023191-175a-4709-9855-e154821a741e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:14:03.682876 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:03.682845    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93023191-175a-4709-9855-e154821a741e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "93023191-175a-4709-9855-e154821a741e" (UID: "93023191-175a-4709-9855-e154821a741e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:14:03.683003 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:03.682953    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93023191-175a-4709-9855-e154821a741e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "93023191-175a-4709-9855-e154821a741e" (UID: "93023191-175a-4709-9855-e154821a741e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:14:03.683003 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:03.682978    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93023191-175a-4709-9855-e154821a741e-kube-api-access-zn7lq" (OuterVolumeSpecName: "kube-api-access-zn7lq") pod "93023191-175a-4709-9855-e154821a741e" (UID: "93023191-175a-4709-9855-e154821a741e"). InnerVolumeSpecName "kube-api-access-zn7lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:03.782181 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:03.782145    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93023191-175a-4709-9855-e154821a741e-openshift-service-ca\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:14:03.782181 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:03.782172    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn7lq\" (UniqueName: \"kubernetes.io/projected/93023191-175a-4709-9855-e154821a741e-kube-api-access-zn7lq\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:14:03.782181 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:03.782182    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93023191-175a-4709-9855-e154821a741e-metrics-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:14:03.782181 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:03.782191    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93023191-175a-4709-9855-e154821a741e-maas-api-tls\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:14:04.289978 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:04.289945    2581 generic.go:356] "Generic (PLEG): container finished" podID="93023191-175a-4709-9855-e154821a741e" containerID="a997db38d9a89b915c69ee7baf0f53f4fc5f062f7b4ce2cc27b129dcc11b3b31" exitCode=0
Sep 10 21:14:04.290160 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:04.289985    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"
Sep 10 21:14:04.290160 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:04.290034    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf" event={"ID":"93023191-175a-4709-9855-e154821a741e","Type":"ContainerDied","Data":"a997db38d9a89b915c69ee7baf0f53f4fc5f062f7b4ce2cc27b129dcc11b3b31"}
Sep 10 21:14:04.290160 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:04.290089    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf" event={"ID":"93023191-175a-4709-9855-e154821a741e","Type":"ContainerDied","Data":"c4abcb3a270c1dd61a981b03797be1b0aae5dc0ea6759d8305857ead7aca8bce"}
Sep 10 21:14:04.290160 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:04.290116    2581 scope.go:117] "RemoveContainer" containerID="a997db38d9a89b915c69ee7baf0f53f4fc5f062f7b4ce2cc27b129dcc11b3b31"
Sep 10 21:14:04.299857 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:04.299842    2581 scope.go:117] "RemoveContainer" containerID="a997db38d9a89b915c69ee7baf0f53f4fc5f062f7b4ce2cc27b129dcc11b3b31"
Sep 10 21:14:04.300142 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:14:04.300125    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a997db38d9a89b915c69ee7baf0f53f4fc5f062f7b4ce2cc27b129dcc11b3b31\": container with ID starting with a997db38d9a89b915c69ee7baf0f53f4fc5f062f7b4ce2cc27b129dcc11b3b31 not found: ID does not exist" containerID="a997db38d9a89b915c69ee7baf0f53f4fc5f062f7b4ce2cc27b129dcc11b3b31"
Sep 10 21:14:04.300239 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:04.300152    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a997db38d9a89b915c69ee7baf0f53f4fc5f062f7b4ce2cc27b129dcc11b3b31"} err="failed to get container status \"a997db38d9a89b915c69ee7baf0f53f4fc5f062f7b4ce2cc27b129dcc11b3b31\": rpc error: code = NotFound desc = could not find container \"a997db38d9a89b915c69ee7baf0f53f4fc5f062f7b4ce2cc27b129dcc11b3b31\": container with ID starting with a997db38d9a89b915c69ee7baf0f53f4fc5f062f7b4ce2cc27b129dcc11b3b31 not found: ID does not exist"
Sep 10 21:14:04.312372 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:04.312347    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"]
Sep 10 21:14:04.316835 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:04.316813    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47f62f-6bc6674ccb-hpwjf"]
Sep 10 21:14:05.574546 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:05.574514    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93023191-175a-4709-9855-e154821a741e" path="/var/lib/kubelet/pods/93023191-175a-4709-9855-e154821a741e/volumes"
Sep 10 21:14:06.694020 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:06.693960    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c" podUID="ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925" containerName="main" containerID="cri-o://590d9a6b53ba874ab4bbf30000ababca945f0b762148ed79c846fbf3654c4fe1" gracePeriod=45
Sep 10 21:14:06.943059 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:06.942997    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:14:07.011532 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.011453    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-tls-certs\") pod \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") "
Sep 10 21:14:07.011532 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.011510    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-home\") pod \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") "
Sep 10 21:14:07.011738 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.011536    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-model-cache\") pod \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") "
Sep 10 21:14:07.011738 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.011626    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-tmp-dir\") pod \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") "
Sep 10 21:14:07.011738 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.011701    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5msb\" (UniqueName: \"kubernetes.io/projected/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-kube-api-access-m5msb\") pod \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") "
Sep 10 21:14:07.011904 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.011742    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-dshm\") pod \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\" (UID: \"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925\") "
Sep 10 21:14:07.011904 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.011755    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-home" (OuterVolumeSpecName: "home") pod "ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925" (UID: "ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:07.011904 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.011835    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-model-cache" (OuterVolumeSpecName: "model-cache") pod "ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925" (UID: "ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:07.011904 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.011897    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925" (UID: "ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:07.012105 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.012024    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-tmp-dir\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:14:07.012105 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.012035    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-home\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:14:07.012105 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.012045    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-model-cache\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:14:07.013933 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.013904    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-kube-api-access-m5msb" (OuterVolumeSpecName: "kube-api-access-m5msb") pod "ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925" (UID: "ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925"). InnerVolumeSpecName "kube-api-access-m5msb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:07.014235 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.014176    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925" (UID: "ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:14:07.014235 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.014176    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-dshm" (OuterVolumeSpecName: "dshm") pod "ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925" (UID: "ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:07.112906 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.112878    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5msb\" (UniqueName: \"kubernetes.io/projected/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-kube-api-access-m5msb\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:14:07.112906 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.112901    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-dshm\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:14:07.112906 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.112912    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925-tls-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:14:07.308337 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.308302    2581 generic.go:356] "Generic (PLEG): container finished" podID="ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925" containerID="590d9a6b53ba874ab4bbf30000ababca945f0b762148ed79c846fbf3654c4fe1" exitCode=0
Sep 10 21:14:07.308554 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.308356    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"
Sep 10 21:14:07.308554 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.308373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c" event={"ID":"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925","Type":"ContainerDied","Data":"590d9a6b53ba874ab4bbf30000ababca945f0b762148ed79c846fbf3654c4fe1"}
Sep 10 21:14:07.308554 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.308404    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c" event={"ID":"ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925","Type":"ContainerDied","Data":"7cc840aa834919f2be24e97e708bf8efe8a1fb4fe0d93e6d3b897b4539478899"}
Sep 10 21:14:07.308554 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.308421    2581 scope.go:117] "RemoveContainer" containerID="590d9a6b53ba874ab4bbf30000ababca945f0b762148ed79c846fbf3654c4fe1"
Sep 10 21:14:07.317129 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.317111    2581 scope.go:117] "RemoveContainer" containerID="590d9a6b53ba874ab4bbf30000ababca945f0b762148ed79c846fbf3654c4fe1"
Sep 10 21:14:07.317433 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:14:07.317412    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590d9a6b53ba874ab4bbf30000ababca945f0b762148ed79c846fbf3654c4fe1\": container with ID starting with 590d9a6b53ba874ab4bbf30000ababca945f0b762148ed79c846fbf3654c4fe1 not found: ID does not exist" containerID="590d9a6b53ba874ab4bbf30000ababca945f0b762148ed79c846fbf3654c4fe1"
Sep 10 21:14:07.317530 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.317440    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590d9a6b53ba874ab4bbf30000ababca945f0b762148ed79c846fbf3654c4fe1"} err="failed to get container status \"590d9a6b53ba874ab4bbf30000ababca945f0b762148ed79c846fbf3654c4fe1\": rpc error: code = NotFound desc = could not find container \"590d9a6b53ba874ab4bbf30000ababca945f0b762148ed79c846fbf3654c4fe1\": container with ID starting with 590d9a6b53ba874ab4bbf30000ababca945f0b762148ed79c846fbf3654c4fe1 not found: ID does not exist"
Sep 10 21:14:07.350908 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.350884    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"]
Sep 10 21:14:07.366100 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.366077    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-7cc2bd/noten-model-7cc2bd-kserve-94b56745f-qmz6c"]
Sep 10 21:14:07.574368 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:07.574287    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925" path="/var/lib/kubelet/pods/ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925/volumes"
Sep 10 21:14:12.470630 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.470596    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"]
Sep 10 21:14:12.471032 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.470973    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93023191-175a-4709-9855-e154821a741e" containerName="maas-api"
Sep 10 21:14:12.471032 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.470984    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="93023191-175a-4709-9855-e154821a741e" containerName="maas-api"
Sep 10 21:14:12.471032 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.471004    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925" containerName="main"
Sep 10 21:14:12.471032 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.471014    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925" containerName="main"
Sep 10 21:14:12.471300 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.471120    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffdb74ac-10d0-4ab7-b0a5-eee66b1f2925" containerName="main"
Sep 10 21:14:12.471300 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.471131    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="93023191-175a-4709-9855-e154821a741e" containerName="maas-api"
Sep 10 21:14:12.475565 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.475549    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"
Sep 10 21:14:12.477847 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.477825    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-db4a68ea\""
Sep 10 21:14:12.478054 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.478039    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-db4a68ea-dockercfg-lpf2s\""
Sep 10 21:14:12.484535 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.484510    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"]
Sep 10 21:14:12.561627 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.561596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2895c0-f574-4d34-aa23-9b430e2023fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm\" (UID: \"9f2895c0-f574-4d34-aa23-9b430e2023fd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"
Sep 10 21:14:12.561801 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.561707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clpz9\" (UniqueName: \"kubernetes.io/projected/9f2895c0-f574-4d34-aa23-9b430e2023fd-kube-api-access-clpz9\") pod \"payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm\" (UID: \"9f2895c0-f574-4d34-aa23-9b430e2023fd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"
Sep 10 21:14:12.662860 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.662821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2895c0-f574-4d34-aa23-9b430e2023fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm\" (UID: \"9f2895c0-f574-4d34-aa23-9b430e2023fd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"
Sep 10 21:14:12.663040 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.662922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clpz9\" (UniqueName: \"kubernetes.io/projected/9f2895c0-f574-4d34-aa23-9b430e2023fd-kube-api-access-clpz9\") pod \"payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm\" (UID: \"9f2895c0-f574-4d34-aa23-9b430e2023fd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"
Sep 10 21:14:12.663668 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.663643    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2895c0-f574-4d34-aa23-9b430e2023fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm\" (UID: \"9f2895c0-f574-4d34-aa23-9b430e2023fd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"
Sep 10 21:14:12.670966 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.670937    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clpz9\" (UniqueName: \"kubernetes.io/projected/9f2895c0-f574-4d34-aa23-9b430e2023fd-kube-api-access-clpz9\") pod \"payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm\" (UID: \"9f2895c0-f574-4d34-aa23-9b430e2023fd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"
Sep 10 21:14:12.785949 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:12.785924    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"
Sep 10 21:14:13.058218 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:13.058179    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"]
Sep 10 21:14:13.060307 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:14:13.060263    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2895c0_f574_4d34_aa23_9b430e2023fd.slice/crio-c3799357707b6d69575eb6aaf861ae2ad72fb7f1d1c2bf63b33ece81d8df7e27 WatchSource:0}: Error finding container c3799357707b6d69575eb6aaf861ae2ad72fb7f1d1c2bf63b33ece81d8df7e27: Status 404 returned error can't find the container with id c3799357707b6d69575eb6aaf861ae2ad72fb7f1d1c2bf63b33ece81d8df7e27
Sep 10 21:14:13.343667 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:13.343584    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm" event={"ID":"9f2895c0-f574-4d34-aa23-9b430e2023fd","Type":"ContainerStarted","Data":"e948cff28120f2b77916f4fa34c6b18df83accbe7418d7a16daacd7158a2e60c"}
Sep 10 21:14:13.343667 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:13.343638    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"
Sep 10 21:14:13.343667 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:13.343656    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm" event={"ID":"9f2895c0-f574-4d34-aa23-9b430e2023fd","Type":"ContainerStarted","Data":"c3799357707b6d69575eb6aaf861ae2ad72fb7f1d1c2bf63b33ece81d8df7e27"}
Sep 10 21:14:13.361184 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:13.361141    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm" podStartSLOduration=1.361126923 podStartE2EDuration="1.361126923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:13.359052228 +0000 UTC m=+994.334712275" watchObservedRunningTime="2026-09-10 21:14:13.361126923 +0000 UTC m=+994.336786970"
Sep 10 21:14:21.777558 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:21.777518    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"]
Sep 10 21:14:21.777994 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:21.777720    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr" podUID="974f55b7-61ea-4f59-8778-306171335bd7" containerName="maas-api" containerID="cri-o://f0f06cc38bd33e37bd3b29e2976eb887859b992b5b0f8045def0f9e7880f389d" gracePeriod=30
Sep 10 21:14:22.223677 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.223647    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:14:22.348186 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.348106    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/974f55b7-61ea-4f59-8778-306171335bd7-metrics-certs\") pod \"974f55b7-61ea-4f59-8778-306171335bd7\" (UID: \"974f55b7-61ea-4f59-8778-306171335bd7\") "
Sep 10 21:14:22.348186 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.348160    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/974f55b7-61ea-4f59-8778-306171335bd7-openshift-service-ca\") pod \"974f55b7-61ea-4f59-8778-306171335bd7\" (UID: \"974f55b7-61ea-4f59-8778-306171335bd7\") "
Sep 10 21:14:22.348186 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.348180    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz9xl\" (UniqueName: \"kubernetes.io/projected/974f55b7-61ea-4f59-8778-306171335bd7-kube-api-access-xz9xl\") pod \"974f55b7-61ea-4f59-8778-306171335bd7\" (UID: \"974f55b7-61ea-4f59-8778-306171335bd7\") "
Sep 10 21:14:22.348483 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.348285    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/974f55b7-61ea-4f59-8778-306171335bd7-maas-api-tls\") pod \"974f55b7-61ea-4f59-8778-306171335bd7\" (UID: \"974f55b7-61ea-4f59-8778-306171335bd7\") "
Sep 10 21:14:22.348549 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.348496    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974f55b7-61ea-4f59-8778-306171335bd7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "974f55b7-61ea-4f59-8778-306171335bd7" (UID: "974f55b7-61ea-4f59-8778-306171335bd7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:14:22.350376 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.350345    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974f55b7-61ea-4f59-8778-306171335bd7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "974f55b7-61ea-4f59-8778-306171335bd7" (UID: "974f55b7-61ea-4f59-8778-306171335bd7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:14:22.350480 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.350388    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974f55b7-61ea-4f59-8778-306171335bd7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "974f55b7-61ea-4f59-8778-306171335bd7" (UID: "974f55b7-61ea-4f59-8778-306171335bd7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:14:22.350480 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.350384    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974f55b7-61ea-4f59-8778-306171335bd7-kube-api-access-xz9xl" (OuterVolumeSpecName: "kube-api-access-xz9xl") pod "974f55b7-61ea-4f59-8778-306171335bd7" (UID: "974f55b7-61ea-4f59-8778-306171335bd7"). InnerVolumeSpecName "kube-api-access-xz9xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:22.389284 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.389252    2581 generic.go:356] "Generic (PLEG): container finished" podID="974f55b7-61ea-4f59-8778-306171335bd7" containerID="f0f06cc38bd33e37bd3b29e2976eb887859b992b5b0f8045def0f9e7880f389d" exitCode=0
Sep 10 21:14:22.389450 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.389309    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr" event={"ID":"974f55b7-61ea-4f59-8778-306171335bd7","Type":"ContainerDied","Data":"f0f06cc38bd33e37bd3b29e2976eb887859b992b5b0f8045def0f9e7880f389d"}
Sep 10 21:14:22.389450 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.389354    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr" event={"ID":"974f55b7-61ea-4f59-8778-306171335bd7","Type":"ContainerDied","Data":"526c2864e9d5afa1a0b56086513abd5425f565763e33ba1cb852421aab331bf3"}
Sep 10 21:14:22.389450 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.389314    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"
Sep 10 21:14:22.389450 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.389384    2581 scope.go:117] "RemoveContainer" containerID="f0f06cc38bd33e37bd3b29e2976eb887859b992b5b0f8045def0f9e7880f389d"
Sep 10 21:14:22.398411 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.398392    2581 scope.go:117] "RemoveContainer" containerID="f0f06cc38bd33e37bd3b29e2976eb887859b992b5b0f8045def0f9e7880f389d"
Sep 10 21:14:22.398661 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:14:22.398647    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f06cc38bd33e37bd3b29e2976eb887859b992b5b0f8045def0f9e7880f389d\": container with ID starting with f0f06cc38bd33e37bd3b29e2976eb887859b992b5b0f8045def0f9e7880f389d not found: ID does not exist" containerID="f0f06cc38bd33e37bd3b29e2976eb887859b992b5b0f8045def0f9e7880f389d"
Sep 10 21:14:22.398707 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.398668    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f06cc38bd33e37bd3b29e2976eb887859b992b5b0f8045def0f9e7880f389d"} err="failed to get container status \"f0f06cc38bd33e37bd3b29e2976eb887859b992b5b0f8045def0f9e7880f389d\": rpc error: code = NotFound desc = could not find container \"f0f06cc38bd33e37bd3b29e2976eb887859b992b5b0f8045def0f9e7880f389d\": container with ID starting with f0f06cc38bd33e37bd3b29e2976eb887859b992b5b0f8045def0f9e7880f389d not found: ID does not exist"
Sep 10 21:14:22.409570 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.409549    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"]
Sep 10 21:14:22.414059 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.414038    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-000295-6dc8f74c9d-fzncr"]
Sep 10 21:14:22.449797 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.449775    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/974f55b7-61ea-4f59-8778-306171335bd7-maas-api-tls\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:14:22.449797 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.449796    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/974f55b7-61ea-4f59-8778-306171335bd7-metrics-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:14:22.450010 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.449806    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/974f55b7-61ea-4f59-8778-306171335bd7-openshift-service-ca\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:14:22.450010 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:22.449815    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz9xl\" (UniqueName: \"kubernetes.io/projected/974f55b7-61ea-4f59-8778-306171335bd7-kube-api-access-xz9xl\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:14:23.573682 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:23.573651    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="974f55b7-61ea-4f59-8778-306171335bd7" path="/var/lib/kubelet/pods/974f55b7-61ea-4f59-8778-306171335bd7/volumes"
Sep 10 21:14:24.350851 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:24.350818    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"
Sep 10 21:14:35.425402 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:35.425368    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"]
Sep 10 21:14:35.425841 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:35.425825    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="974f55b7-61ea-4f59-8778-306171335bd7" containerName="maas-api"
Sep 10 21:14:35.425924 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:35.425843    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="974f55b7-61ea-4f59-8778-306171335bd7" containerName="maas-api"
Sep 10 21:14:35.425963 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:35.425938    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="974f55b7-61ea-4f59-8778-306171335bd7" containerName="maas-api"
Sep 10 21:14:35.430286 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:35.430270    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"
Sep 10 21:14:35.432292 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:35.432265    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-828c7c\""
Sep 10 21:14:35.432401 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:35.432334    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-828c7c-dockercfg-lkwzv\""
Sep 10 21:14:35.440903 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:35.440879    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"]
Sep 10 21:14:35.474175 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:35.474143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a840edb-409e-4e7e-95ff-1bd29a81f614-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89\" (UID: \"3a840edb-409e-4e7e-95ff-1bd29a81f614\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"
Sep 10 21:14:35.474355 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:35.474243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxlvq\" (UniqueName: \"kubernetes.io/projected/3a840edb-409e-4e7e-95ff-1bd29a81f614-kube-api-access-xxlvq\") pod \"payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89\" (UID: \"3a840edb-409e-4e7e-95ff-1bd29a81f614\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"
Sep 10 21:14:35.575175 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:35.575142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a840edb-409e-4e7e-95ff-1bd29a81f614-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89\" (UID: \"3a840edb-409e-4e7e-95ff-1bd29a81f614\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"
Sep 10 21:14:35.575352 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:35.575197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxlvq\" (UniqueName: \"kubernetes.io/projected/3a840edb-409e-4e7e-95ff-1bd29a81f614-kube-api-access-xxlvq\") pod \"payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89\" (UID: \"3a840edb-409e-4e7e-95ff-1bd29a81f614\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"
Sep 10 21:14:35.575770 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:35.575749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a840edb-409e-4e7e-95ff-1bd29a81f614-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89\" (UID: \"3a840edb-409e-4e7e-95ff-1bd29a81f614\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"
Sep 10 21:14:35.583674 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:35.583652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxlvq\" (UniqueName: \"kubernetes.io/projected/3a840edb-409e-4e7e-95ff-1bd29a81f614-kube-api-access-xxlvq\") pod \"payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89\" (UID: \"3a840edb-409e-4e7e-95ff-1bd29a81f614\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"
Sep 10 21:14:35.741961 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:35.741889    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"
Sep 10 21:14:36.012025 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:36.011860    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"]
Sep 10 21:14:36.014287 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:14:36.014260    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a840edb_409e_4e7e_95ff_1bd29a81f614.slice/crio-3fe85cd62c6a1d7ae51cc3c1e7b3238a382ad2bb526f8578c9b2f8168c5c877c WatchSource:0}: Error finding container 3fe85cd62c6a1d7ae51cc3c1e7b3238a382ad2bb526f8578c9b2f8168c5c877c: Status 404 returned error can't find the container with id 3fe85cd62c6a1d7ae51cc3c1e7b3238a382ad2bb526f8578c9b2f8168c5c877c
Sep 10 21:14:36.461778 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:36.461745    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89" event={"ID":"3a840edb-409e-4e7e-95ff-1bd29a81f614","Type":"ContainerStarted","Data":"eee843771ff737ab852e6f47277f28878f8b45af86c8be4eeee98f3a80efbee7"}
Sep 10 21:14:36.462148 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:36.461823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89" event={"ID":"3a840edb-409e-4e7e-95ff-1bd29a81f614","Type":"ContainerStarted","Data":"3fe85cd62c6a1d7ae51cc3c1e7b3238a382ad2bb526f8578c9b2f8168c5c877c"}
Sep 10 21:14:36.462148 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:36.461868    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"
Sep 10 21:14:36.481804 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:36.481756    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89" podStartSLOduration=1.481742135 podStartE2EDuration="1.481742135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:36.480712109 +0000 UTC m=+1017.456372157" watchObservedRunningTime="2026-09-10 21:14:36.481742135 +0000 UTC m=+1017.457402219"
Sep 10 21:14:47.469643 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:47.469608    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"
Sep 10 21:14:53.485892 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:53.485853    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"]
Sep 10 21:14:53.486474 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:53.486115    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm" podUID="9f2895c0-f574-4d34-aa23-9b430e2023fd" containerName="payload-pre-processing" containerID="cri-o://e948cff28120f2b77916f4fa34c6b18df83accbe7418d7a16daacd7158a2e60c" gracePeriod=30
Sep 10 21:14:53.722611 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:53.722585    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"
Sep 10 21:14:53.837129 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:53.837103    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clpz9\" (UniqueName: \"kubernetes.io/projected/9f2895c0-f574-4d34-aa23-9b430e2023fd-kube-api-access-clpz9\") pod \"9f2895c0-f574-4d34-aa23-9b430e2023fd\" (UID: \"9f2895c0-f574-4d34-aa23-9b430e2023fd\") "
Sep 10 21:14:53.837339 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:53.837174    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2895c0-f574-4d34-aa23-9b430e2023fd-plugins-config-volume\") pod \"9f2895c0-f574-4d34-aa23-9b430e2023fd\" (UID: \"9f2895c0-f574-4d34-aa23-9b430e2023fd\") "
Sep 10 21:14:53.837493 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:53.837468    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2895c0-f574-4d34-aa23-9b430e2023fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f2895c0-f574-4d34-aa23-9b430e2023fd" (UID: "9f2895c0-f574-4d34-aa23-9b430e2023fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:14:53.839056 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:53.839036    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2895c0-f574-4d34-aa23-9b430e2023fd-kube-api-access-clpz9" (OuterVolumeSpecName: "kube-api-access-clpz9") pod "9f2895c0-f574-4d34-aa23-9b430e2023fd" (UID: "9f2895c0-f574-4d34-aa23-9b430e2023fd"). InnerVolumeSpecName "kube-api-access-clpz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:53.938361 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:53.938325    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clpz9\" (UniqueName: \"kubernetes.io/projected/9f2895c0-f574-4d34-aa23-9b430e2023fd-kube-api-access-clpz9\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:14:53.938361 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:53.938354    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2895c0-f574-4d34-aa23-9b430e2023fd-plugins-config-volume\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:14:54.549906 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:54.549868    2581 generic.go:356] "Generic (PLEG): container finished" podID="9f2895c0-f574-4d34-aa23-9b430e2023fd" containerID="e948cff28120f2b77916f4fa34c6b18df83accbe7418d7a16daacd7158a2e60c" exitCode=0
Sep 10 21:14:54.550344 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:54.549921    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"
Sep 10 21:14:54.550344 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:54.549939    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm" event={"ID":"9f2895c0-f574-4d34-aa23-9b430e2023fd","Type":"ContainerDied","Data":"e948cff28120f2b77916f4fa34c6b18df83accbe7418d7a16daacd7158a2e60c"}
Sep 10 21:14:54.550344 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:54.549994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm" event={"ID":"9f2895c0-f574-4d34-aa23-9b430e2023fd","Type":"ContainerDied","Data":"c3799357707b6d69575eb6aaf861ae2ad72fb7f1d1c2bf63b33ece81d8df7e27"}
Sep 10 21:14:54.550344 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:54.550023    2581 scope.go:117] "RemoveContainer" containerID="e948cff28120f2b77916f4fa34c6b18df83accbe7418d7a16daacd7158a2e60c"
Sep 10 21:14:54.558817 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:54.558796    2581 scope.go:117] "RemoveContainer" containerID="e948cff28120f2b77916f4fa34c6b18df83accbe7418d7a16daacd7158a2e60c"
Sep 10 21:14:54.559082 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:14:54.559065    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e948cff28120f2b77916f4fa34c6b18df83accbe7418d7a16daacd7158a2e60c\": container with ID starting with e948cff28120f2b77916f4fa34c6b18df83accbe7418d7a16daacd7158a2e60c not found: ID does not exist" containerID="e948cff28120f2b77916f4fa34c6b18df83accbe7418d7a16daacd7158a2e60c"
Sep 10 21:14:54.559130 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:54.559089    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e948cff28120f2b77916f4fa34c6b18df83accbe7418d7a16daacd7158a2e60c"} err="failed to get container status \"e948cff28120f2b77916f4fa34c6b18df83accbe7418d7a16daacd7158a2e60c\": rpc error: code = NotFound desc = could not find container \"e948cff28120f2b77916f4fa34c6b18df83accbe7418d7a16daacd7158a2e60c\": container with ID starting with e948cff28120f2b77916f4fa34c6b18df83accbe7418d7a16daacd7158a2e60c not found: ID does not exist"
Sep 10 21:14:54.571494 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:54.571469    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"]
Sep 10 21:14:54.575732 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:54.575710    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db4a68ea-77fc69df5d-n27pm"]
Sep 10 21:14:55.573460 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:14:55.573426    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f2895c0-f574-4d34-aa23-9b430e2023fd" path="/var/lib/kubelet/pods/9f2895c0-f574-4d34-aa23-9b430e2023fd/volumes"
Sep 10 21:15:19.035515 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.035483    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"]
Sep 10 21:15:19.035987 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.035939    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f2895c0-f574-4d34-aa23-9b430e2023fd" containerName="payload-pre-processing"
Sep 10 21:15:19.035987 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.035956    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f2895c0-f574-4d34-aa23-9b430e2023fd" containerName="payload-pre-processing"
Sep 10 21:15:19.036104 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.036064    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f2895c0-f574-4d34-aa23-9b430e2023fd" containerName="payload-pre-processing"
Sep 10 21:15:19.039188 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.039170    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:15:19.041450 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.041430    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-14cee3-serving-cert\""
Sep 10 21:15:19.049516 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.049491    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"]
Sep 10 21:15:19.078144 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.078115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk\" (UID: \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:15:19.078311 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.078182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk\" (UID: \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:15:19.078311 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.078224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmdt8\" (UniqueName: \"kubernetes.io/projected/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-kube-api-access-fmdt8\") pod \"maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk\" (UID: \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:15:19.078311 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.078256    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk\" (UID: \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:15:19.179289 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.179252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk\" (UID: \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:15:19.179444 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.179301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmdt8\" (UniqueName: \"kubernetes.io/projected/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-kube-api-access-fmdt8\") pod \"maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk\" (UID: \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:15:19.179444 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.179344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk\" (UID: \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:15:19.179444 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.179410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk\" (UID: \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:15:19.179612 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:15:19.179555    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-serving-cert: secret "maas-api-e2e-shared-a-w2-14cee3-serving-cert" not found
Sep 10 21:15:19.179668 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:15:19.179632    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-maas-api-tls podName:2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36 nodeName:}" failed. No retries permitted until 2026-09-10 21:15:19.679611095 +0000 UTC m=+1060.655271137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-maas-api-tls") pod "maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk" (UID: "2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36") : secret "maas-api-e2e-shared-a-w2-14cee3-serving-cert" not found
Sep 10 21:15:19.179863 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.179841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk\" (UID: \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:15:19.181910 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.181890    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk\" (UID: \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:15:19.187740 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.187713    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmdt8\" (UniqueName: \"kubernetes.io/projected/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-kube-api-access-fmdt8\") pod \"maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk\" (UID: \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:15:19.683877 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.683846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk\" (UID: \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:15:19.686165 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.686148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk\" (UID: \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:15:19.951564 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:19.951489    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:15:20.222455 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:20.222424    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"]
Sep 10 21:15:20.223800 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:15:20.223774    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf0e4a7_594b_4706_af6b_2c2bf6cd2b36.slice/crio-d29d00a8b649b2143db70d92d1163ca85bdea480074f2decbcb630f477cd6bb6 WatchSource:0}: Error finding container d29d00a8b649b2143db70d92d1163ca85bdea480074f2decbcb630f477cd6bb6: Status 404 returned error can't find the container with id d29d00a8b649b2143db70d92d1163ca85bdea480074f2decbcb630f477cd6bb6
Sep 10 21:15:20.225230 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:20.225187    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:15:20.678248 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:20.678197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk" event={"ID":"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36","Type":"ContainerStarted","Data":"414e138079689246a257d1f455d0b6da2eaccf5ced34e1bab1606e4a4b2a9e9a"}
Sep 10 21:15:20.678384 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:20.678270    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk" event={"ID":"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36","Type":"ContainerStarted","Data":"d29d00a8b649b2143db70d92d1163ca85bdea480074f2decbcb630f477cd6bb6"}
Sep 10 21:15:20.678384 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:20.678292    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:15:20.698320 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:20.698273    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk" podStartSLOduration=1.332605405 podStartE2EDuration="1.698256882s" totalImagesPullingTime="365.651477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:20.696846635 +0000 UTC m=+1061.672506683" watchObservedRunningTime="2026-09-10 21:15:20.698256882 +0000 UTC m=+1061.673917331"
Sep 10 21:15:26.687977 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:26.687948    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:15:41.401420 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.401383    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"]
Sep 10 21:15:41.405961 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.405939    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.408928 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.408903    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-828c7c\"/\"openshift-service-ca.crt\""
Sep 10 21:15:41.409072 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.408985    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-828c7c\"/\"kube-root-ca.crt\""
Sep 10 21:15:41.409072 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.408942    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-828c7c\"/\"auth-test-model-828c7c-kserve-self-signed-certs\""
Sep 10 21:15:41.409072 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.408942    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-828c7c\"/\"default-dockercfg-5bcln\""
Sep 10 21:15:41.415000 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.414976    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"]
Sep 10 21:15:41.479856 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.479825    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-home\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.479987 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.479866    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-model-cache\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.479987 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.479897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-dshm\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.479987 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.479923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7ddb\" (UniqueName: \"kubernetes.io/projected/ecb65078-4047-4970-8d24-8fdf022f8cfc-kube-api-access-z7ddb\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.479987 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.479961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb65078-4047-4970-8d24-8fdf022f8cfc-tls-certs\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.480125 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.480021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-tmp-dir\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.581438 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.581413    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-home\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.581595 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.581450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-model-cache\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.581595 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.581480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-dshm\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.581595 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.581523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7ddb\" (UniqueName: \"kubernetes.io/projected/ecb65078-4047-4970-8d24-8fdf022f8cfc-kube-api-access-z7ddb\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.581595 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.581585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb65078-4047-4970-8d24-8fdf022f8cfc-tls-certs\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.581794 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.581638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-tmp-dir\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.581903 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.581883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-model-cache\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.581963 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.581915    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-home\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.581963 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.581952    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-tmp-dir\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.583542 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.583524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-dshm\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.584005 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.583981    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb65078-4047-4970-8d24-8fdf022f8cfc-tls-certs\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.590101 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.590080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7ddb\" (UniqueName: \"kubernetes.io/projected/ecb65078-4047-4970-8d24-8fdf022f8cfc-kube-api-access-z7ddb\") pod \"auth-test-model-828c7c-kserve-798cb746dc-bhs26\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.716605 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.716536    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:41.989376 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:41.989352    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"]
Sep 10 21:15:41.991258 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:15:41.991227    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecb65078_4047_4970_8d24_8fdf022f8cfc.slice/crio-e1c854edef02e08060f0a6714c3fbb679fbd61de213fb6df862104f029d1584d WatchSource:0}: Error finding container e1c854edef02e08060f0a6714c3fbb679fbd61de213fb6df862104f029d1584d: Status 404 returned error can't find the container with id e1c854edef02e08060f0a6714c3fbb679fbd61de213fb6df862104f029d1584d
Sep 10 21:15:42.786581 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:42.786549    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26" event={"ID":"ecb65078-4047-4970-8d24-8fdf022f8cfc","Type":"ContainerStarted","Data":"ffcc38df8446d2262de5504741172c76e59d4a242dfe56cb6585fa5e93b168e6"}
Sep 10 21:15:42.786964 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:42.786588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26" event={"ID":"ecb65078-4047-4970-8d24-8fdf022f8cfc","Type":"ContainerStarted","Data":"e1c854edef02e08060f0a6714c3fbb679fbd61de213fb6df862104f029d1584d"}
Sep 10 21:15:42.805152 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:42.805109    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26" podStartSLOduration=1.8050958719999999 podStartE2EDuration="1.805095872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:42.803357175 +0000 UTC m=+1083.779017221" watchObservedRunningTime="2026-09-10 21:15:42.805095872 +0000 UTC m=+1083.780755949"
Sep 10 21:15:48.668298 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:48.668261    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"]
Sep 10 21:15:48.671829 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:48.671814    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"
Sep 10 21:15:48.675136 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:48.675114    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-aca18f-dockercfg-fdht9\""
Sep 10 21:15:48.675576 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:48.675560    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-aca18f\""
Sep 10 21:15:48.695251 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:48.695227    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"]
Sep 10 21:15:48.753911 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:48.753882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvrs2\" (UniqueName: \"kubernetes.io/projected/cce8decc-17da-409f-8af6-c94255909688-kube-api-access-mvrs2\") pod \"payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6\" (UID: \"cce8decc-17da-409f-8af6-c94255909688\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"
Sep 10 21:15:48.754058 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:48.753986    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce8decc-17da-409f-8af6-c94255909688-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6\" (UID: \"cce8decc-17da-409f-8af6-c94255909688\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"
Sep 10 21:15:48.854928 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:48.854897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce8decc-17da-409f-8af6-c94255909688-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6\" (UID: \"cce8decc-17da-409f-8af6-c94255909688\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"
Sep 10 21:15:48.855070 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:48.854977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvrs2\" (UniqueName: \"kubernetes.io/projected/cce8decc-17da-409f-8af6-c94255909688-kube-api-access-mvrs2\") pod \"payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6\" (UID: \"cce8decc-17da-409f-8af6-c94255909688\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"
Sep 10 21:15:48.855586 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:48.855569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce8decc-17da-409f-8af6-c94255909688-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6\" (UID: \"cce8decc-17da-409f-8af6-c94255909688\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"
Sep 10 21:15:48.865363 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:48.865341    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvrs2\" (UniqueName: \"kubernetes.io/projected/cce8decc-17da-409f-8af6-c94255909688-kube-api-access-mvrs2\") pod \"payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6\" (UID: \"cce8decc-17da-409f-8af6-c94255909688\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"
Sep 10 21:15:48.983446 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:48.983370    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"
Sep 10 21:15:49.250075 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:49.250051    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"]
Sep 10 21:15:49.252447 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:15:49.252418    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce8decc_17da_409f_8af6_c94255909688.slice/crio-83d5ad90aad1f2f08ae017c4ca0dc0de86e19f681d0944cda17d98790c4fb87a WatchSource:0}: Error finding container 83d5ad90aad1f2f08ae017c4ca0dc0de86e19f681d0944cda17d98790c4fb87a: Status 404 returned error can't find the container with id 83d5ad90aad1f2f08ae017c4ca0dc0de86e19f681d0944cda17d98790c4fb87a
Sep 10 21:15:49.824364 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:49.824323    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6" event={"ID":"cce8decc-17da-409f-8af6-c94255909688","Type":"ContainerStarted","Data":"7fcc73cdf6ddcddb1e2b4e5e0a31ee63e3bc8dd52e32cb26295f3e26274de5fc"}
Sep 10 21:15:49.824832 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:49.824424    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6" event={"ID":"cce8decc-17da-409f-8af6-c94255909688","Type":"ContainerStarted","Data":"83d5ad90aad1f2f08ae017c4ca0dc0de86e19f681d0944cda17d98790c4fb87a"}
Sep 10 21:15:49.824832 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:49.824448    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"
Sep 10 21:15:49.843029 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:49.842986    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6" podStartSLOduration=1.842973238 podStartE2EDuration="1.842973238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:49.840978428 +0000 UTC m=+1090.816638478" watchObservedRunningTime="2026-09-10 21:15:49.842973238 +0000 UTC m=+1090.818633284"
Sep 10 21:15:51.716694 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:51.716654    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:51.716694 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:51.716702    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:51.729113 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:51.729085    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:15:51.845419 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:15:51.845383    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:16:00.831574 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:00.831539    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"
Sep 10 21:16:14.600935 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.600898    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"]
Sep 10 21:16:14.604658 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.604638    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.607020 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.606997    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9f6475\"/\"openshift-service-ca.crt\""
Sep 10 21:16:14.608018 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.608002    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9f6475\"/\"kube-root-ca.crt\""
Sep 10 21:16:14.608125 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.608018    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9f6475\"/\"default-dockercfg-m4fdt\""
Sep 10 21:16:14.608125 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.608031    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9f6475\"/\"auth-test-model-9f6475-kserve-self-signed-certs\""
Sep 10 21:16:14.615097 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.615076    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"]
Sep 10 21:16:14.688342 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.688311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-home\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.688506 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.688400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zljdp\" (UniqueName: \"kubernetes.io/projected/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-kube-api-access-zljdp\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.688506 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.688441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-dshm\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.688506 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.688499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-tmp-dir\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.688650 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.688526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-tls-certs\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.688650 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.688562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-model-cache\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.789720 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.789696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-tls-certs\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.789864 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.789734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-model-cache\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.789864 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.789767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-home\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.789864 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.789831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljdp\" (UniqueName: \"kubernetes.io/projected/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-kube-api-access-zljdp\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.790016 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.789971    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-dshm\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.790070 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.790035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-tmp-dir\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.790180 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.790158    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-home\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.790295 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.790178    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-model-cache\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.790428 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.790408    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-tmp-dir\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.792139 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.792111    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-dshm\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.792386 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.792369    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-tls-certs\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.798052 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.798024    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zljdp\" (UniqueName: \"kubernetes.io/projected/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-kube-api-access-zljdp\") pod \"auth-test-model-9f6475-kserve-59769b8675-cjqv8\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:14.914382 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:14.914298    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:15.197039 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:15.197016    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"]
Sep 10 21:16:15.198903 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:16:15.198876    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod063c80d9_ec17_40bc_97aa_c8f7fca9c41a.slice/crio-a863c808533b2c59cf8ae6c07a9dc7edec73744360d4c6b439a59cced11271cc WatchSource:0}: Error finding container a863c808533b2c59cf8ae6c07a9dc7edec73744360d4c6b439a59cced11271cc: Status 404 returned error can't find the container with id a863c808533b2c59cf8ae6c07a9dc7edec73744360d4c6b439a59cced11271cc
Sep 10 21:16:15.955143 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:15.955108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8" event={"ID":"063c80d9-ec17-40bc-97aa-c8f7fca9c41a","Type":"ContainerStarted","Data":"04969e727e0ea1cb3ad018d12b51bf07932a4d5b3fd5da7a04c9736220578e1d"}
Sep 10 21:16:15.955143 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:15.955148    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8" event={"ID":"063c80d9-ec17-40bc-97aa-c8f7fca9c41a","Type":"ContainerStarted","Data":"a863c808533b2c59cf8ae6c07a9dc7edec73744360d4c6b439a59cced11271cc"}
Sep 10 21:16:15.971913 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:15.971867    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8" podStartSLOduration=1.971852131 podStartE2EDuration="1.971852131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:15.970852998 +0000 UTC m=+1116.946513045" watchObservedRunningTime="2026-09-10 21:16:15.971852131 +0000 UTC m=+1116.947512177"
Sep 10 21:16:24.914655 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:24.914618    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:24.915049 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:24.914895    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:24.928012 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:24.927986    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:25.009806 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:25.009780    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:16:58.993486 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:58.993453    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"]
Sep 10 21:16:58.997273 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:58.997253    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.002742 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.002717    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-669d5f22\"/\"openshift-service-ca.crt\""
Sep 10 21:16:59.003682 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.003561    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-669d5f22\"/\"e2e-lifecycle-model-669d5f22-kserve-self-signed-certs\""
Sep 10 21:16:59.003682 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.003604    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-669d5f22\"/\"default-dockercfg-mf6x9\""
Sep 10 21:16:59.004073 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.004058    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-669d5f22\"/\"kube-root-ca.crt\""
Sep 10 21:16:59.008581 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.008561    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"]
Sep 10 21:16:59.105366 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.105335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-home\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.105550 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.105390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-tmp-dir\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.105550 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.105415    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-model-cache\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.105550 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.105464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-dshm\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.105550 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.105510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2173ae82-42e8-4b83-b25e-ea09384099c7-tls-certs\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.105725 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.105561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qtst\" (UniqueName: \"kubernetes.io/projected/2173ae82-42e8-4b83-b25e-ea09384099c7-kube-api-access-9qtst\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.206121 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.206081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-dshm\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.206121 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.206122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2173ae82-42e8-4b83-b25e-ea09384099c7-tls-certs\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.206404 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.206172    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qtst\" (UniqueName: \"kubernetes.io/projected/2173ae82-42e8-4b83-b25e-ea09384099c7-kube-api-access-9qtst\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.206404 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.206278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-home\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.206404 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.206348    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-tmp-dir\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.206404 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.206390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-model-cache\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.206638 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.206615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-home\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.206717 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.206694    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-tmp-dir\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.206816 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.206801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-model-cache\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.208242 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.208224    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-dshm\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.208510 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.208492    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2173ae82-42e8-4b83-b25e-ea09384099c7-tls-certs\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.216126 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.216103    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qtst\" (UniqueName: \"kubernetes.io/projected/2173ae82-42e8-4b83-b25e-ea09384099c7-kube-api-access-9qtst\") pod \"e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") " pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.307963 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.307931    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:16:59.581682 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:16:59.581658    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"]
Sep 10 21:16:59.583098 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:16:59.583069    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2173ae82_42e8_4b83_b25e_ea09384099c7.slice/crio-8cf4ab787fcc4fb7f7684884f339870b3160f10263ac63b6169e849016d5c4f1 WatchSource:0}: Error finding container 8cf4ab787fcc4fb7f7684884f339870b3160f10263ac63b6169e849016d5c4f1: Status 404 returned error can't find the container with id 8cf4ab787fcc4fb7f7684884f339870b3160f10263ac63b6169e849016d5c4f1
Sep 10 21:17:00.181065 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:17:00.181023    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv" event={"ID":"2173ae82-42e8-4b83-b25e-ea09384099c7","Type":"ContainerStarted","Data":"b55e9f1e74651537f9be2baad65f8b755a3c5b44f3e9a519b82fc3cd8dee678d"}
Sep 10 21:17:00.181451 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:17:00.181076    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv" event={"ID":"2173ae82-42e8-4b83-b25e-ea09384099c7","Type":"ContainerStarted","Data":"8cf4ab787fcc4fb7f7684884f339870b3160f10263ac63b6169e849016d5c4f1"}
Sep 10 21:17:00.201606 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:17:00.201429    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv" podStartSLOduration=2.201410573 podStartE2EDuration="2.201410573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:00.201017815 +0000 UTC m=+1161.176677863" watchObservedRunningTime="2026-09-10 21:17:00.201410573 +0000 UTC m=+1161.177070621"
Sep 10 21:17:09.308342 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:17:09.308306    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:17:09.308811 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:17:09.308374    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:17:09.320685 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:17:09.320663    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:17:10.246610 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:17:10.246582    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:18:03.732608 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:03.732575    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"]
Sep 10 21:18:18.752684 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:18.752575    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv" podUID="2173ae82-42e8-4b83-b25e-ea09384099c7" containerName="main" containerID="cri-o://b55e9f1e74651537f9be2baad65f8b755a3c5b44f3e9a519b82fc3cd8dee678d" gracePeriod=45
Sep 10 21:18:19.003138 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.003074    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:18:19.050807 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.050781    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2173ae82-42e8-4b83-b25e-ea09384099c7-tls-certs\") pod \"2173ae82-42e8-4b83-b25e-ea09384099c7\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") "
Sep 10 21:18:19.050980 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.050831    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-model-cache\") pod \"2173ae82-42e8-4b83-b25e-ea09384099c7\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") "
Sep 10 21:18:19.050980 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.050852    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-dshm\") pod \"2173ae82-42e8-4b83-b25e-ea09384099c7\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") "
Sep 10 21:18:19.050980 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.050877    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-tmp-dir\") pod \"2173ae82-42e8-4b83-b25e-ea09384099c7\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") "
Sep 10 21:18:19.050980 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.050920    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qtst\" (UniqueName: \"kubernetes.io/projected/2173ae82-42e8-4b83-b25e-ea09384099c7-kube-api-access-9qtst\") pod \"2173ae82-42e8-4b83-b25e-ea09384099c7\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") "
Sep 10 21:18:19.050980 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.050956    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-home\") pod \"2173ae82-42e8-4b83-b25e-ea09384099c7\" (UID: \"2173ae82-42e8-4b83-b25e-ea09384099c7\") "
Sep 10 21:18:19.051281 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.051113    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "2173ae82-42e8-4b83-b25e-ea09384099c7" (UID: "2173ae82-42e8-4b83-b25e-ea09384099c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:19.051368 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.051349    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-model-cache\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:19.051429 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.051345    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-home" (OuterVolumeSpecName: "home") pod "2173ae82-42e8-4b83-b25e-ea09384099c7" (UID: "2173ae82-42e8-4b83-b25e-ea09384099c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:19.051429 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.051396    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2173ae82-42e8-4b83-b25e-ea09384099c7" (UID: "2173ae82-42e8-4b83-b25e-ea09384099c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:19.053371 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.053337    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2173ae82-42e8-4b83-b25e-ea09384099c7-kube-api-access-9qtst" (OuterVolumeSpecName: "kube-api-access-9qtst") pod "2173ae82-42e8-4b83-b25e-ea09384099c7" (UID: "2173ae82-42e8-4b83-b25e-ea09384099c7"). InnerVolumeSpecName "kube-api-access-9qtst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:18:19.053500 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.053406    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2173ae82-42e8-4b83-b25e-ea09384099c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2173ae82-42e8-4b83-b25e-ea09384099c7" (UID: "2173ae82-42e8-4b83-b25e-ea09384099c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:18:19.053500 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.053411    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-dshm" (OuterVolumeSpecName: "dshm") pod "2173ae82-42e8-4b83-b25e-ea09384099c7" (UID: "2173ae82-42e8-4b83-b25e-ea09384099c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:19.152495 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.152466    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2173ae82-42e8-4b83-b25e-ea09384099c7-tls-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:19.152495 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.152487    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-dshm\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:19.152495 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.152496    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-tmp-dir\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:19.152750 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.152504    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qtst\" (UniqueName: \"kubernetes.io/projected/2173ae82-42e8-4b83-b25e-ea09384099c7-kube-api-access-9qtst\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:19.152750 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.152515    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2173ae82-42e8-4b83-b25e-ea09384099c7-home\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:19.587103 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.587069    2581 generic.go:356] "Generic (PLEG): container finished" podID="2173ae82-42e8-4b83-b25e-ea09384099c7" containerID="b55e9f1e74651537f9be2baad65f8b755a3c5b44f3e9a519b82fc3cd8dee678d" exitCode=0
Sep 10 21:18:19.587258 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.587113    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"
Sep 10 21:18:19.587258 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.587131    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv" event={"ID":"2173ae82-42e8-4b83-b25e-ea09384099c7","Type":"ContainerDied","Data":"b55e9f1e74651537f9be2baad65f8b755a3c5b44f3e9a519b82fc3cd8dee678d"}
Sep 10 21:18:19.587258 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.587160    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv" event={"ID":"2173ae82-42e8-4b83-b25e-ea09384099c7","Type":"ContainerDied","Data":"8cf4ab787fcc4fb7f7684884f339870b3160f10263ac63b6169e849016d5c4f1"}
Sep 10 21:18:19.587258 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.587176    2581 scope.go:117] "RemoveContainer" containerID="b55e9f1e74651537f9be2baad65f8b755a3c5b44f3e9a519b82fc3cd8dee678d"
Sep 10 21:18:19.595897 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.595879    2581 scope.go:117] "RemoveContainer" containerID="b55e9f1e74651537f9be2baad65f8b755a3c5b44f3e9a519b82fc3cd8dee678d"
Sep 10 21:18:19.596165 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:18:19.596143    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55e9f1e74651537f9be2baad65f8b755a3c5b44f3e9a519b82fc3cd8dee678d\": container with ID starting with b55e9f1e74651537f9be2baad65f8b755a3c5b44f3e9a519b82fc3cd8dee678d not found: ID does not exist" containerID="b55e9f1e74651537f9be2baad65f8b755a3c5b44f3e9a519b82fc3cd8dee678d"
Sep 10 21:18:19.596226 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.596174    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55e9f1e74651537f9be2baad65f8b755a3c5b44f3e9a519b82fc3cd8dee678d"} err="failed to get container status \"b55e9f1e74651537f9be2baad65f8b755a3c5b44f3e9a519b82fc3cd8dee678d\": rpc error: code = NotFound desc = could not find container \"b55e9f1e74651537f9be2baad65f8b755a3c5b44f3e9a519b82fc3cd8dee678d\": container with ID starting with b55e9f1e74651537f9be2baad65f8b755a3c5b44f3e9a519b82fc3cd8dee678d not found: ID does not exist"
Sep 10 21:18:19.603644 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.603620    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"]
Sep 10 21:18:19.607127 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:19.607103    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-669d5f22/e2e-lifecycle-model-669d5f22-kserve-858984664c-ssdqv"]
Sep 10 21:18:21.574452 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:21.574415    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2173ae82-42e8-4b83-b25e-ea09384099c7" path="/var/lib/kubelet/pods/2173ae82-42e8-4b83-b25e-ea09384099c7/volumes"
Sep 10 21:18:36.808832 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:36.808752    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"]
Sep 10 21:18:37.921358 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:37.921329    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"]
Sep 10 21:18:51.828245 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:51.828170    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26" podUID="ecb65078-4047-4970-8d24-8fdf022f8cfc" containerName="main" containerID="cri-o://ffcc38df8446d2262de5504741172c76e59d4a242dfe56cb6585fa5e93b168e6" gracePeriod=45
Sep 10 21:18:51.834916 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:51.834883    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26" podUID="ecb65078-4047-4970-8d24-8fdf022f8cfc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 10 21:18:52.078981 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.078924    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:18:52.270387 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.270353    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-dshm\") pod \"ecb65078-4047-4970-8d24-8fdf022f8cfc\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") "
Sep 10 21:18:52.270650 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.270412    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7ddb\" (UniqueName: \"kubernetes.io/projected/ecb65078-4047-4970-8d24-8fdf022f8cfc-kube-api-access-z7ddb\") pod \"ecb65078-4047-4970-8d24-8fdf022f8cfc\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") "
Sep 10 21:18:52.270650 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.270447    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-model-cache\") pod \"ecb65078-4047-4970-8d24-8fdf022f8cfc\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") "
Sep 10 21:18:52.270650 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.270477    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-tmp-dir\") pod \"ecb65078-4047-4970-8d24-8fdf022f8cfc\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") "
Sep 10 21:18:52.270650 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.270499    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-home\") pod \"ecb65078-4047-4970-8d24-8fdf022f8cfc\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") "
Sep 10 21:18:52.270650 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.270533    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb65078-4047-4970-8d24-8fdf022f8cfc-tls-certs\") pod \"ecb65078-4047-4970-8d24-8fdf022f8cfc\" (UID: \"ecb65078-4047-4970-8d24-8fdf022f8cfc\") "
Sep 10 21:18:52.270928 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.270725    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-model-cache" (OuterVolumeSpecName: "model-cache") pod "ecb65078-4047-4970-8d24-8fdf022f8cfc" (UID: "ecb65078-4047-4970-8d24-8fdf022f8cfc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:52.270928 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.270772    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ecb65078-4047-4970-8d24-8fdf022f8cfc" (UID: "ecb65078-4047-4970-8d24-8fdf022f8cfc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:52.270928 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.270793    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-home" (OuterVolumeSpecName: "home") pod "ecb65078-4047-4970-8d24-8fdf022f8cfc" (UID: "ecb65078-4047-4970-8d24-8fdf022f8cfc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:52.271083 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.271013    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-model-cache\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:52.271083 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.271058    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-tmp-dir\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:52.271083 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.271073    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-home\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:52.272655 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.272627    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-dshm" (OuterVolumeSpecName: "dshm") pod "ecb65078-4047-4970-8d24-8fdf022f8cfc" (UID: "ecb65078-4047-4970-8d24-8fdf022f8cfc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:52.272761 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.272706    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb65078-4047-4970-8d24-8fdf022f8cfc-kube-api-access-z7ddb" (OuterVolumeSpecName: "kube-api-access-z7ddb") pod "ecb65078-4047-4970-8d24-8fdf022f8cfc" (UID: "ecb65078-4047-4970-8d24-8fdf022f8cfc"). InnerVolumeSpecName "kube-api-access-z7ddb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:18:52.272807 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.272756    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb65078-4047-4970-8d24-8fdf022f8cfc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ecb65078-4047-4970-8d24-8fdf022f8cfc" (UID: "ecb65078-4047-4970-8d24-8fdf022f8cfc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:18:52.371660 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.371602    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7ddb\" (UniqueName: \"kubernetes.io/projected/ecb65078-4047-4970-8d24-8fdf022f8cfc-kube-api-access-z7ddb\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:52.371660 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.371623    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb65078-4047-4970-8d24-8fdf022f8cfc-tls-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:52.371660 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.371632    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb65078-4047-4970-8d24-8fdf022f8cfc-dshm\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:52.747069 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.746987    2581 generic.go:356] "Generic (PLEG): container finished" podID="ecb65078-4047-4970-8d24-8fdf022f8cfc" containerID="ffcc38df8446d2262de5504741172c76e59d4a242dfe56cb6585fa5e93b168e6" exitCode=0
Sep 10 21:18:52.747069 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.747024    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"
Sep 10 21:18:52.747069 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.747049    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26" event={"ID":"ecb65078-4047-4970-8d24-8fdf022f8cfc","Type":"ContainerDied","Data":"ffcc38df8446d2262de5504741172c76e59d4a242dfe56cb6585fa5e93b168e6"}
Sep 10 21:18:52.747350 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.747079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26" event={"ID":"ecb65078-4047-4970-8d24-8fdf022f8cfc","Type":"ContainerDied","Data":"e1c854edef02e08060f0a6714c3fbb679fbd61de213fb6df862104f029d1584d"}
Sep 10 21:18:52.747350 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.747095    2581 scope.go:117] "RemoveContainer" containerID="ffcc38df8446d2262de5504741172c76e59d4a242dfe56cb6585fa5e93b168e6"
Sep 10 21:18:52.756093 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.756074    2581 scope.go:117] "RemoveContainer" containerID="ffcc38df8446d2262de5504741172c76e59d4a242dfe56cb6585fa5e93b168e6"
Sep 10 21:18:52.756380 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:18:52.756362    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffcc38df8446d2262de5504741172c76e59d4a242dfe56cb6585fa5e93b168e6\": container with ID starting with ffcc38df8446d2262de5504741172c76e59d4a242dfe56cb6585fa5e93b168e6 not found: ID does not exist" containerID="ffcc38df8446d2262de5504741172c76e59d4a242dfe56cb6585fa5e93b168e6"
Sep 10 21:18:52.756432 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.756388    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffcc38df8446d2262de5504741172c76e59d4a242dfe56cb6585fa5e93b168e6"} err="failed to get container status \"ffcc38df8446d2262de5504741172c76e59d4a242dfe56cb6585fa5e93b168e6\": rpc error: code = NotFound desc = could not find container \"ffcc38df8446d2262de5504741172c76e59d4a242dfe56cb6585fa5e93b168e6\": container with ID starting with ffcc38df8446d2262de5504741172c76e59d4a242dfe56cb6585fa5e93b168e6 not found: ID does not exist"
Sep 10 21:18:52.767367 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.767347    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"]
Sep 10 21:18:52.770471 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.770450    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-828c7c/auth-test-model-828c7c-kserve-798cb746dc-bhs26"]
Sep 10 21:18:52.940961 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:52.940925    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8" podUID="063c80d9-ec17-40bc-97aa-c8f7fca9c41a" containerName="main" containerID="cri-o://04969e727e0ea1cb3ad018d12b51bf07932a4d5b3fd5da7a04c9736220578e1d" gracePeriod=45
Sep 10 21:18:53.195099 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.195078    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:18:53.280153 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.280122    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-dshm\") pod \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") "
Sep 10 21:18:53.280312 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.280191    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-tmp-dir\") pod \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") "
Sep 10 21:18:53.280312 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.280242    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-tls-certs\") pod \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") "
Sep 10 21:18:53.280312 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.280279    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-home\") pod \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") "
Sep 10 21:18:53.280466 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.280328    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zljdp\" (UniqueName: \"kubernetes.io/projected/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-kube-api-access-zljdp\") pod \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") "
Sep 10 21:18:53.280466 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.280401    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-model-cache\") pod \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\" (UID: \"063c80d9-ec17-40bc-97aa-c8f7fca9c41a\") "
Sep 10 21:18:53.280585 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.280468    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "063c80d9-ec17-40bc-97aa-c8f7fca9c41a" (UID: "063c80d9-ec17-40bc-97aa-c8f7fca9c41a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:53.280694 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.280665    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-model-cache" (OuterVolumeSpecName: "model-cache") pod "063c80d9-ec17-40bc-97aa-c8f7fca9c41a" (UID: "063c80d9-ec17-40bc-97aa-c8f7fca9c41a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:53.280832 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.280700    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-home" (OuterVolumeSpecName: "home") pod "063c80d9-ec17-40bc-97aa-c8f7fca9c41a" (UID: "063c80d9-ec17-40bc-97aa-c8f7fca9c41a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:53.280832 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.280769    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-model-cache\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:53.280832 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.280787    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-tmp-dir\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:53.282557 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.282534    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-kube-api-access-zljdp" (OuterVolumeSpecName: "kube-api-access-zljdp") pod "063c80d9-ec17-40bc-97aa-c8f7fca9c41a" (UID: "063c80d9-ec17-40bc-97aa-c8f7fca9c41a"). InnerVolumeSpecName "kube-api-access-zljdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:18:53.282743 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.282729    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-dshm" (OuterVolumeSpecName: "dshm") pod "063c80d9-ec17-40bc-97aa-c8f7fca9c41a" (UID: "063c80d9-ec17-40bc-97aa-c8f7fca9c41a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:53.282805 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.282749    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "063c80d9-ec17-40bc-97aa-c8f7fca9c41a" (UID: "063c80d9-ec17-40bc-97aa-c8f7fca9c41a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:18:53.381517 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.381456    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-dshm\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:53.381517 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.381477    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-tls-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:53.381517 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.381486    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-home\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:53.381517 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.381494    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zljdp\" (UniqueName: \"kubernetes.io/projected/063c80d9-ec17-40bc-97aa-c8f7fca9c41a-kube-api-access-zljdp\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:18:53.576358 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.576321    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecb65078-4047-4970-8d24-8fdf022f8cfc" path="/var/lib/kubelet/pods/ecb65078-4047-4970-8d24-8fdf022f8cfc/volumes"
Sep 10 21:18:53.753710 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.753619    2581 generic.go:356] "Generic (PLEG): container finished" podID="063c80d9-ec17-40bc-97aa-c8f7fca9c41a" containerID="04969e727e0ea1cb3ad018d12b51bf07932a4d5b3fd5da7a04c9736220578e1d" exitCode=0
Sep 10 21:18:53.753710 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.753658    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"
Sep 10 21:18:53.753951 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.753658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8" event={"ID":"063c80d9-ec17-40bc-97aa-c8f7fca9c41a","Type":"ContainerDied","Data":"04969e727e0ea1cb3ad018d12b51bf07932a4d5b3fd5da7a04c9736220578e1d"}
Sep 10 21:18:53.753951 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.753757    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8" event={"ID":"063c80d9-ec17-40bc-97aa-c8f7fca9c41a","Type":"ContainerDied","Data":"a863c808533b2c59cf8ae6c07a9dc7edec73744360d4c6b439a59cced11271cc"}
Sep 10 21:18:53.753951 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.753780    2581 scope.go:117] "RemoveContainer" containerID="04969e727e0ea1cb3ad018d12b51bf07932a4d5b3fd5da7a04c9736220578e1d"
Sep 10 21:18:53.762045 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.762014    2581 scope.go:117] "RemoveContainer" containerID="04969e727e0ea1cb3ad018d12b51bf07932a4d5b3fd5da7a04c9736220578e1d"
Sep 10 21:18:53.762320 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:18:53.762302    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04969e727e0ea1cb3ad018d12b51bf07932a4d5b3fd5da7a04c9736220578e1d\": container with ID starting with 04969e727e0ea1cb3ad018d12b51bf07932a4d5b3fd5da7a04c9736220578e1d not found: ID does not exist" containerID="04969e727e0ea1cb3ad018d12b51bf07932a4d5b3fd5da7a04c9736220578e1d"
Sep 10 21:18:53.762384 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.762326    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04969e727e0ea1cb3ad018d12b51bf07932a4d5b3fd5da7a04c9736220578e1d"} err="failed to get container status \"04969e727e0ea1cb3ad018d12b51bf07932a4d5b3fd5da7a04c9736220578e1d\": rpc error: code = NotFound desc = could not find container \"04969e727e0ea1cb3ad018d12b51bf07932a4d5b3fd5da7a04c9736220578e1d\": container with ID starting with 04969e727e0ea1cb3ad018d12b51bf07932a4d5b3fd5da7a04c9736220578e1d not found: ID does not exist"
Sep 10 21:18:53.768364 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.768343    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"]
Sep 10 21:18:53.772262 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:53.772240    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/auth-test-model-9f6475-kserve-59769b8675-cjqv8"]
Sep 10 21:18:55.574495 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:18:55.574465    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="063c80d9-ec17-40bc-97aa-c8f7fca9c41a" path="/var/lib/kubelet/pods/063c80d9-ec17-40bc-97aa-c8f7fca9c41a/volumes"
Sep 10 21:19:17.704955 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.704922    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"]
Sep 10 21:19:17.705446 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.705367    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2173ae82-42e8-4b83-b25e-ea09384099c7" containerName="main"
Sep 10 21:19:17.705446 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.705383    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2173ae82-42e8-4b83-b25e-ea09384099c7" containerName="main"
Sep 10 21:19:17.705446 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.705406    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="063c80d9-ec17-40bc-97aa-c8f7fca9c41a" containerName="main"
Sep 10 21:19:17.705446 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.705416    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="063c80d9-ec17-40bc-97aa-c8f7fca9c41a" containerName="main"
Sep 10 21:19:17.705446 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.705446    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb65078-4047-4970-8d24-8fdf022f8cfc" containerName="main"
Sep 10 21:19:17.705692 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.705455    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb65078-4047-4970-8d24-8fdf022f8cfc" containerName="main"
Sep 10 21:19:17.705692 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.705553    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecb65078-4047-4970-8d24-8fdf022f8cfc" containerName="main"
Sep 10 21:19:17.705692 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.705564    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2173ae82-42e8-4b83-b25e-ea09384099c7" containerName="main"
Sep 10 21:19:17.705692 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.705578    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="063c80d9-ec17-40bc-97aa-c8f7fca9c41a" containerName="main"
Sep 10 21:19:17.709315 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.709295    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.712303 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.712094    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9f6475\"/\"default-dockercfg-m4fdt\""
Sep 10 21:19:17.712303 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.712101    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9f6475\"/\"openshift-service-ca.crt\""
Sep 10 21:19:17.712303 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.712113    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9f6475\"/\"kube-root-ca.crt\""
Sep 10 21:19:17.712303 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.712242    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9f6475\"/\"sub-test-model-9f6475-kserve-self-signed-certs\""
Sep 10 21:19:17.719036 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.719012    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"]
Sep 10 21:19:17.807535 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.807506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp2qf\" (UniqueName: \"kubernetes.io/projected/b005ba92-158d-4048-9fae-74c9fada952d-kube-api-access-dp2qf\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.807535 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.807537    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-home\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.807738 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.807559    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b005ba92-158d-4048-9fae-74c9fada952d-tls-certs\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.807738 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.807626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-model-cache\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.807738 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.807727    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-dshm\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.807904 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.807802    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-tmp-dir\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.908739 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.908704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-tmp-dir\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.908931 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.908778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp2qf\" (UniqueName: \"kubernetes.io/projected/b005ba92-158d-4048-9fae-74c9fada952d-kube-api-access-dp2qf\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.908931 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.908796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-home\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.908931 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.908810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b005ba92-158d-4048-9fae-74c9fada952d-tls-certs\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.908931 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.908832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-model-cache\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.908931 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.908882    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-dshm\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.909247 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.909089    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-tmp-dir\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.909247 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.909150    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-home\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.909247 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.909226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-model-cache\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.911185 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.911158    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-dshm\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.911499 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.911479    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b005ba92-158d-4048-9fae-74c9fada952d-tls-certs\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:17.917541 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:17.917516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp2qf\" (UniqueName: \"kubernetes.io/projected/b005ba92-158d-4048-9fae-74c9fada952d-kube-api-access-dp2qf\") pod \"sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:18.019991 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:18.019910    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:18.298892 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:18.295755    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"]
Sep 10 21:19:18.301890 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:19:18.301861    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb005ba92_158d_4048_9fae_74c9fada952d.slice/crio-980777a53b083557223f011d8c6f8830038bdf0c88d04c2ff98e460cce4c5a80 WatchSource:0}: Error finding container 980777a53b083557223f011d8c6f8830038bdf0c88d04c2ff98e460cce4c5a80: Status 404 returned error can't find the container with id 980777a53b083557223f011d8c6f8830038bdf0c88d04c2ff98e460cce4c5a80
Sep 10 21:19:18.883383 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:18.883349    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j" event={"ID":"b005ba92-158d-4048-9fae-74c9fada952d","Type":"ContainerStarted","Data":"6d92f6c026a1c618e62b89f38839d05e2fa8237ee9a3f05b512c5c3eb12e8719"}
Sep 10 21:19:18.883810 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:18.883397    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j" event={"ID":"b005ba92-158d-4048-9fae-74c9fada952d","Type":"ContainerStarted","Data":"980777a53b083557223f011d8c6f8830038bdf0c88d04c2ff98e460cce4c5a80"}
Sep 10 21:19:18.901047 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:18.900999    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j" podStartSLOduration=1.900983032 podStartE2EDuration="1.900983032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:18.899970458 +0000 UTC m=+1299.875630518" watchObservedRunningTime="2026-09-10 21:19:18.900983032 +0000 UTC m=+1299.876643133"
Sep 10 21:19:28.020490 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:28.020459    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:28.021004 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:28.020506    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:28.033657 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:28.033632    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:28.943034 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:28.943003    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:19:53.366970 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.366934    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"]
Sep 10 21:19:53.375692 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.375671    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:19:53.378796 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.378766    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0778a4-serving-cert\""
Sep 10 21:19:53.385432 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.385410    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"]
Sep 10 21:19:53.449632 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.449601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf6pg\" (UniqueName: \"kubernetes.io/projected/cde04da8-fedf-409c-adfc-fdd286e665d5-kube-api-access-nf6pg\") pod \"maas-api-e2e-api-a-0778a4-567775cb9c-twd58\" (UID: \"cde04da8-fedf-409c-adfc-fdd286e665d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:19:53.449760 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.449670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde04da8-fedf-409c-adfc-fdd286e665d5-openshift-service-ca\") pod \"maas-api-e2e-api-a-0778a4-567775cb9c-twd58\" (UID: \"cde04da8-fedf-409c-adfc-fdd286e665d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:19:53.449805 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.449766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde04da8-fedf-409c-adfc-fdd286e665d5-metrics-certs\") pod \"maas-api-e2e-api-a-0778a4-567775cb9c-twd58\" (UID: \"cde04da8-fedf-409c-adfc-fdd286e665d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:19:53.449805 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.449798    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cde04da8-fedf-409c-adfc-fdd286e665d5-maas-api-tls\") pod \"maas-api-e2e-api-a-0778a4-567775cb9c-twd58\" (UID: \"cde04da8-fedf-409c-adfc-fdd286e665d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:19:53.550773 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.550740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6pg\" (UniqueName: \"kubernetes.io/projected/cde04da8-fedf-409c-adfc-fdd286e665d5-kube-api-access-nf6pg\") pod \"maas-api-e2e-api-a-0778a4-567775cb9c-twd58\" (UID: \"cde04da8-fedf-409c-adfc-fdd286e665d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:19:53.550920 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.550794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde04da8-fedf-409c-adfc-fdd286e665d5-openshift-service-ca\") pod \"maas-api-e2e-api-a-0778a4-567775cb9c-twd58\" (UID: \"cde04da8-fedf-409c-adfc-fdd286e665d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:19:53.550920 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.550842    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde04da8-fedf-409c-adfc-fdd286e665d5-metrics-certs\") pod \"maas-api-e2e-api-a-0778a4-567775cb9c-twd58\" (UID: \"cde04da8-fedf-409c-adfc-fdd286e665d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:19:53.550920 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.550869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cde04da8-fedf-409c-adfc-fdd286e665d5-maas-api-tls\") pod \"maas-api-e2e-api-a-0778a4-567775cb9c-twd58\" (UID: \"cde04da8-fedf-409c-adfc-fdd286e665d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:19:53.551624 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.551603    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde04da8-fedf-409c-adfc-fdd286e665d5-openshift-service-ca\") pod \"maas-api-e2e-api-a-0778a4-567775cb9c-twd58\" (UID: \"cde04da8-fedf-409c-adfc-fdd286e665d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:19:53.553266 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.553244    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cde04da8-fedf-409c-adfc-fdd286e665d5-maas-api-tls\") pod \"maas-api-e2e-api-a-0778a4-567775cb9c-twd58\" (UID: \"cde04da8-fedf-409c-adfc-fdd286e665d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:19:53.553387 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.553366    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde04da8-fedf-409c-adfc-fdd286e665d5-metrics-certs\") pod \"maas-api-e2e-api-a-0778a4-567775cb9c-twd58\" (UID: \"cde04da8-fedf-409c-adfc-fdd286e665d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:19:53.558309 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.558282    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf6pg\" (UniqueName: \"kubernetes.io/projected/cde04da8-fedf-409c-adfc-fdd286e665d5-kube-api-access-nf6pg\") pod \"maas-api-e2e-api-a-0778a4-567775cb9c-twd58\" (UID: \"cde04da8-fedf-409c-adfc-fdd286e665d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:19:53.687773 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.687689    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:19:53.961064 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:53.961041    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"]
Sep 10 21:19:53.963101 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:19:53.963074    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde04da8_fedf_409c_adfc_fdd286e665d5.slice/crio-b06e23dcd2be2b4435ac0a66f5f17f877263db91e66252e996f3ae9f0489867a WatchSource:0}: Error finding container b06e23dcd2be2b4435ac0a66f5f17f877263db91e66252e996f3ae9f0489867a: Status 404 returned error can't find the container with id b06e23dcd2be2b4435ac0a66f5f17f877263db91e66252e996f3ae9f0489867a
Sep 10 21:19:54.007701 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:54.007677    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"]
Sep 10 21:19:54.012505 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:54.012482    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"
Sep 10 21:19:54.014795 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:54.014773    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0778a4\""
Sep 10 21:19:54.014915 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:54.014856    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0778a4-dockercfg-4g7j6\""
Sep 10 21:19:54.021180 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:54.021144    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"]
Sep 10 21:19:54.053893 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:54.053861    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58" event={"ID":"cde04da8-fedf-409c-adfc-fdd286e665d5","Type":"ContainerStarted","Data":"b06e23dcd2be2b4435ac0a66f5f17f877263db91e66252e996f3ae9f0489867a"}
Sep 10 21:19:54.157630 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:54.157596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jz75\" (UniqueName: \"kubernetes.io/projected/9dc7d43e-143b-46ad-bfcc-1a421c67a67d-kube-api-access-6jz75\") pod \"payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c\" (UID: \"9dc7d43e-143b-46ad-bfcc-1a421c67a67d\") " pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"
Sep 10 21:19:54.157782 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:54.157712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc7d43e-143b-46ad-bfcc-1a421c67a67d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c\" (UID: \"9dc7d43e-143b-46ad-bfcc-1a421c67a67d\") " pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"
Sep 10 21:19:54.259103 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:54.259011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc7d43e-143b-46ad-bfcc-1a421c67a67d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c\" (UID: \"9dc7d43e-143b-46ad-bfcc-1a421c67a67d\") " pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"
Sep 10 21:19:54.259103 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:54.259085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jz75\" (UniqueName: \"kubernetes.io/projected/9dc7d43e-143b-46ad-bfcc-1a421c67a67d-kube-api-access-6jz75\") pod \"payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c\" (UID: \"9dc7d43e-143b-46ad-bfcc-1a421c67a67d\") " pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"
Sep 10 21:19:54.259791 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:54.259765    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc7d43e-143b-46ad-bfcc-1a421c67a67d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c\" (UID: \"9dc7d43e-143b-46ad-bfcc-1a421c67a67d\") " pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"
Sep 10 21:19:54.266850 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:54.266825    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jz75\" (UniqueName: \"kubernetes.io/projected/9dc7d43e-143b-46ad-bfcc-1a421c67a67d-kube-api-access-6jz75\") pod \"payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c\" (UID: \"9dc7d43e-143b-46ad-bfcc-1a421c67a67d\") " pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"
Sep 10 21:19:54.325423 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:54.325398    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"
Sep 10 21:19:54.596967 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:54.596943    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"]
Sep 10 21:19:54.598954 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:19:54.598932    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc7d43e_143b_46ad_bfcc_1a421c67a67d.slice/crio-a52020f84ed32b3e68eb5d2bf686330bfb826a818a0727764c1aa8ea690a084a WatchSource:0}: Error finding container a52020f84ed32b3e68eb5d2bf686330bfb826a818a0727764c1aa8ea690a084a: Status 404 returned error can't find the container with id a52020f84ed32b3e68eb5d2bf686330bfb826a818a0727764c1aa8ea690a084a
Sep 10 21:19:55.061937 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:55.061899    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c" event={"ID":"9dc7d43e-143b-46ad-bfcc-1a421c67a67d","Type":"ContainerStarted","Data":"647070bf35a17069880bcb481c16b13b1a5d340934f5bf64b36cb25d3a37fdfa"}
Sep 10 21:19:55.062112 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:55.061964    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"
Sep 10 21:19:55.062112 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:55.061990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c" event={"ID":"9dc7d43e-143b-46ad-bfcc-1a421c67a67d","Type":"ContainerStarted","Data":"a52020f84ed32b3e68eb5d2bf686330bfb826a818a0727764c1aa8ea690a084a"}
Sep 10 21:19:55.063516 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:55.063492    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58" event={"ID":"cde04da8-fedf-409c-adfc-fdd286e665d5","Type":"ContainerStarted","Data":"a7ee49893981b3919bfea01937e8df1071175e830ac96c26f9deaf9272ea958e"}
Sep 10 21:19:55.063828 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:55.063591    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:19:55.080804 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:55.080763    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c" podStartSLOduration=2.080750754 podStartE2EDuration="2.080750754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:55.079307116 +0000 UTC m=+1336.054967164" watchObservedRunningTime="2026-09-10 21:19:55.080750754 +0000 UTC m=+1336.056410843"
Sep 10 21:19:55.096053 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:55.096012    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58" podStartSLOduration=1.597793898 podStartE2EDuration="2.095995587s" totalImagesPullingTime="498.201689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:55.095292636 +0000 UTC m=+1336.070952684" watchObservedRunningTime="2026-09-10 21:19:55.095995587 +0000 UTC m=+1336.071655635"
Sep 10 21:19:56.889165 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:19:56.889124    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"]
Sep 10 21:20:01.074300 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:01.074265    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:20:06.070419 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:06.070340    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"
Sep 10 21:20:11.909606 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:11.909565    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j" podUID="b005ba92-158d-4048-9fae-74c9fada952d" containerName="main" containerID="cri-o://6d92f6c026a1c618e62b89f38839d05e2fa8237ee9a3f05b512c5c3eb12e8719" gracePeriod=45
Sep 10 21:20:12.149510 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.149476    2581 generic.go:356] "Generic (PLEG): container finished" podID="b005ba92-158d-4048-9fae-74c9fada952d" containerID="6d92f6c026a1c618e62b89f38839d05e2fa8237ee9a3f05b512c5c3eb12e8719" exitCode=0
Sep 10 21:20:12.149642 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.149515    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j" event={"ID":"b005ba92-158d-4048-9fae-74c9fada952d","Type":"ContainerDied","Data":"6d92f6c026a1c618e62b89f38839d05e2fa8237ee9a3f05b512c5c3eb12e8719"}
Sep 10 21:20:12.149704 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.149639    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j" event={"ID":"b005ba92-158d-4048-9fae-74c9fada952d","Type":"ContainerDied","Data":"980777a53b083557223f011d8c6f8830038bdf0c88d04c2ff98e460cce4c5a80"}
Sep 10 21:20:12.149704 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.149682    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="980777a53b083557223f011d8c6f8830038bdf0c88d04c2ff98e460cce4c5a80"
Sep 10 21:20:12.165043 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.164988    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:20:12.323161 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.323127    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp2qf\" (UniqueName: \"kubernetes.io/projected/b005ba92-158d-4048-9fae-74c9fada952d-kube-api-access-dp2qf\") pod \"b005ba92-158d-4048-9fae-74c9fada952d\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") "
Sep 10 21:20:12.323351 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.323181    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b005ba92-158d-4048-9fae-74c9fada952d-tls-certs\") pod \"b005ba92-158d-4048-9fae-74c9fada952d\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") "
Sep 10 21:20:12.323351 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.323233    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-tmp-dir\") pod \"b005ba92-158d-4048-9fae-74c9fada952d\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") "
Sep 10 21:20:12.323351 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.323283    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-home\") pod \"b005ba92-158d-4048-9fae-74c9fada952d\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") "
Sep 10 21:20:12.323351 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.323326    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-dshm\") pod \"b005ba92-158d-4048-9fae-74c9fada952d\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") "
Sep 10 21:20:12.323591 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.323385    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-model-cache\") pod \"b005ba92-158d-4048-9fae-74c9fada952d\" (UID: \"b005ba92-158d-4048-9fae-74c9fada952d\") "
Sep 10 21:20:12.323591 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.323517    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b005ba92-158d-4048-9fae-74c9fada952d" (UID: "b005ba92-158d-4048-9fae-74c9fada952d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:12.323702 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.323650    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-home" (OuterVolumeSpecName: "home") pod "b005ba92-158d-4048-9fae-74c9fada952d" (UID: "b005ba92-158d-4048-9fae-74c9fada952d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:12.323702 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.323677    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-model-cache" (OuterVolumeSpecName: "model-cache") pod "b005ba92-158d-4048-9fae-74c9fada952d" (UID: "b005ba92-158d-4048-9fae-74c9fada952d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:12.324053 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.324035    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-tmp-dir\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:20:12.324138 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.324054    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-home\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:20:12.324138 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.324066    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-model-cache\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:20:12.325346 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.325321    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-dshm" (OuterVolumeSpecName: "dshm") pod "b005ba92-158d-4048-9fae-74c9fada952d" (UID: "b005ba92-158d-4048-9fae-74c9fada952d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:12.325435 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.325346    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b005ba92-158d-4048-9fae-74c9fada952d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b005ba92-158d-4048-9fae-74c9fada952d" (UID: "b005ba92-158d-4048-9fae-74c9fada952d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:20:12.325478 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.325445    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b005ba92-158d-4048-9fae-74c9fada952d-kube-api-access-dp2qf" (OuterVolumeSpecName: "kube-api-access-dp2qf") pod "b005ba92-158d-4048-9fae-74c9fada952d" (UID: "b005ba92-158d-4048-9fae-74c9fada952d"). InnerVolumeSpecName "kube-api-access-dp2qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:20:12.424891 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.424816    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp2qf\" (UniqueName: \"kubernetes.io/projected/b005ba92-158d-4048-9fae-74c9fada952d-kube-api-access-dp2qf\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:20:12.424891 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.424840    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b005ba92-158d-4048-9fae-74c9fada952d-tls-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:20:12.424891 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:12.424850    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b005ba92-158d-4048-9fae-74c9fada952d-dshm\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:20:13.154700 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:13.154672    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"
Sep 10 21:20:13.176809 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:13.176785    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"]
Sep 10 21:20:13.180986 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:13.180964    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/sub-test-model-9f6475-kserve-7d6f55c5b5-zj99j"]
Sep 10 21:20:13.574013 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:13.573982    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b005ba92-158d-4048-9fae-74c9fada952d" path="/var/lib/kubelet/pods/b005ba92-158d-4048-9fae-74c9fada952d/volumes"
Sep 10 21:20:22.939544 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:22.939509    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"]
Sep 10 21:20:22.940161 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:22.940141    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b005ba92-158d-4048-9fae-74c9fada952d" containerName="main"
Sep 10 21:20:22.940161 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:22.940163    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b005ba92-158d-4048-9fae-74c9fada952d" containerName="main"
Sep 10 21:20:22.940332 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:22.940309    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b005ba92-158d-4048-9fae-74c9fada952d" containerName="main"
Sep 10 21:20:22.945102 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:22.945079    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:20:22.947632 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:22.947612    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-806a33-serving-cert\""
Sep 10 21:20:22.954802 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:22.954775    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"]
Sep 10 21:20:23.014817 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.014787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vcds\" (UniqueName: \"kubernetes.io/projected/34ee329b-33c0-47a6-a8b6-442b9d266252-kube-api-access-7vcds\") pod \"maas-api-e2e-api-b-806a33-6545679d48-rfbtf\" (UID: \"34ee329b-33c0-47a6-a8b6-442b9d266252\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:20:23.014955 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.014850    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ee329b-33c0-47a6-a8b6-442b9d266252-maas-api-tls\") pod \"maas-api-e2e-api-b-806a33-6545679d48-rfbtf\" (UID: \"34ee329b-33c0-47a6-a8b6-442b9d266252\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:20:23.014955 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.014895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ee329b-33c0-47a6-a8b6-442b9d266252-openshift-service-ca\") pod \"maas-api-e2e-api-b-806a33-6545679d48-rfbtf\" (UID: \"34ee329b-33c0-47a6-a8b6-442b9d266252\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:20:23.014955 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.014929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ee329b-33c0-47a6-a8b6-442b9d266252-metrics-certs\") pod \"maas-api-e2e-api-b-806a33-6545679d48-rfbtf\" (UID: \"34ee329b-33c0-47a6-a8b6-442b9d266252\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:20:23.115764 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.115724    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ee329b-33c0-47a6-a8b6-442b9d266252-metrics-certs\") pod \"maas-api-e2e-api-b-806a33-6545679d48-rfbtf\" (UID: \"34ee329b-33c0-47a6-a8b6-442b9d266252\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:20:23.115971 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.115799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vcds\" (UniqueName: \"kubernetes.io/projected/34ee329b-33c0-47a6-a8b6-442b9d266252-kube-api-access-7vcds\") pod \"maas-api-e2e-api-b-806a33-6545679d48-rfbtf\" (UID: \"34ee329b-33c0-47a6-a8b6-442b9d266252\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:20:23.115971 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.115872    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ee329b-33c0-47a6-a8b6-442b9d266252-maas-api-tls\") pod \"maas-api-e2e-api-b-806a33-6545679d48-rfbtf\" (UID: \"34ee329b-33c0-47a6-a8b6-442b9d266252\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:20:23.115971 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.115919    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ee329b-33c0-47a6-a8b6-442b9d266252-openshift-service-ca\") pod \"maas-api-e2e-api-b-806a33-6545679d48-rfbtf\" (UID: \"34ee329b-33c0-47a6-a8b6-442b9d266252\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:20:23.115971 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:20:23.115947    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-serving-cert: secret "maas-api-e2e-api-b-806a33-serving-cert" not found
Sep 10 21:20:23.116191 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:20:23.116031    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34ee329b-33c0-47a6-a8b6-442b9d266252-maas-api-tls podName:34ee329b-33c0-47a6-a8b6-442b9d266252 nodeName:}" failed. No retries permitted until 2026-09-10 21:20:23.616009493 +0000 UTC m=+1364.591669523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34ee329b-33c0-47a6-a8b6-442b9d266252-maas-api-tls") pod "maas-api-e2e-api-b-806a33-6545679d48-rfbtf" (UID: "34ee329b-33c0-47a6-a8b6-442b9d266252") : secret "maas-api-e2e-api-b-806a33-serving-cert" not found
Sep 10 21:20:23.116679 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.116659    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ee329b-33c0-47a6-a8b6-442b9d266252-openshift-service-ca\") pod \"maas-api-e2e-api-b-806a33-6545679d48-rfbtf\" (UID: \"34ee329b-33c0-47a6-a8b6-442b9d266252\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:20:23.118177 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.118157    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ee329b-33c0-47a6-a8b6-442b9d266252-metrics-certs\") pod \"maas-api-e2e-api-b-806a33-6545679d48-rfbtf\" (UID: \"34ee329b-33c0-47a6-a8b6-442b9d266252\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:20:23.123956 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.123933    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vcds\" (UniqueName: \"kubernetes.io/projected/34ee329b-33c0-47a6-a8b6-442b9d266252-kube-api-access-7vcds\") pod \"maas-api-e2e-api-b-806a33-6545679d48-rfbtf\" (UID: \"34ee329b-33c0-47a6-a8b6-442b9d266252\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:20:23.573635 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.573607    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"]
Sep 10 21:20:23.577404 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.577379    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"
Sep 10 21:20:23.579581 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.579555    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-806a33\""
Sep 10 21:20:23.584388 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.584364    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"]
Sep 10 21:20:23.620628 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.620595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtr4w\" (UniqueName: \"kubernetes.io/projected/5baa9611-8f58-4257-8bdb-7ae8480003bc-kube-api-access-gtr4w\") pod \"payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n\" (UID: \"5baa9611-8f58-4257-8bdb-7ae8480003bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"
Sep 10 21:20:23.620793 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.620696    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5baa9611-8f58-4257-8bdb-7ae8480003bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n\" (UID: \"5baa9611-8f58-4257-8bdb-7ae8480003bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"
Sep 10 21:20:23.620866 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.620822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ee329b-33c0-47a6-a8b6-442b9d266252-maas-api-tls\") pod \"maas-api-e2e-api-b-806a33-6545679d48-rfbtf\" (UID: \"34ee329b-33c0-47a6-a8b6-442b9d266252\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:20:23.623334 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.623310    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ee329b-33c0-47a6-a8b6-442b9d266252-maas-api-tls\") pod \"maas-api-e2e-api-b-806a33-6545679d48-rfbtf\" (UID: \"34ee329b-33c0-47a6-a8b6-442b9d266252\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:20:23.722275 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.722242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtr4w\" (UniqueName: \"kubernetes.io/projected/5baa9611-8f58-4257-8bdb-7ae8480003bc-kube-api-access-gtr4w\") pod \"payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n\" (UID: \"5baa9611-8f58-4257-8bdb-7ae8480003bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"
Sep 10 21:20:23.722446 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.722299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5baa9611-8f58-4257-8bdb-7ae8480003bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n\" (UID: \"5baa9611-8f58-4257-8bdb-7ae8480003bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"
Sep 10 21:20:23.722988 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.722966    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5baa9611-8f58-4257-8bdb-7ae8480003bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n\" (UID: \"5baa9611-8f58-4257-8bdb-7ae8480003bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"
Sep 10 21:20:23.730742 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.730717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtr4w\" (UniqueName: \"kubernetes.io/projected/5baa9611-8f58-4257-8bdb-7ae8480003bc-kube-api-access-gtr4w\") pod \"payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n\" (UID: \"5baa9611-8f58-4257-8bdb-7ae8480003bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"
Sep 10 21:20:23.859003 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.858928    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:20:23.890049 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:23.890020    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"
Sep 10 21:20:24.131577 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:24.131550    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"]
Sep 10 21:20:24.133457 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:20:24.133428    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ee329b_33c0_47a6_a8b6_442b9d266252.slice/crio-61bd5afc7a951aff13379f15683f014cb449f3c711f14c25f19c7828b3d6dc1e WatchSource:0}: Error finding container 61bd5afc7a951aff13379f15683f014cb449f3c711f14c25f19c7828b3d6dc1e: Status 404 returned error can't find the container with id 61bd5afc7a951aff13379f15683f014cb449f3c711f14c25f19c7828b3d6dc1e
Sep 10 21:20:24.134871 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:24.134850    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:20:24.166758 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:24.166737    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"]
Sep 10 21:20:24.168690 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:20:24.168668    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5baa9611_8f58_4257_8bdb_7ae8480003bc.slice/crio-cd8d768144196aec81a46eea32c538516964a5ca75394e661823f4914958a222 WatchSource:0}: Error finding container cd8d768144196aec81a46eea32c538516964a5ca75394e661823f4914958a222: Status 404 returned error can't find the container with id cd8d768144196aec81a46eea32c538516964a5ca75394e661823f4914958a222
Sep 10 21:20:24.208574 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:24.208545    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n" event={"ID":"5baa9611-8f58-4257-8bdb-7ae8480003bc","Type":"ContainerStarted","Data":"cd8d768144196aec81a46eea32c538516964a5ca75394e661823f4914958a222"}
Sep 10 21:20:24.209772 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:24.209746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf" event={"ID":"34ee329b-33c0-47a6-a8b6-442b9d266252","Type":"ContainerStarted","Data":"61bd5afc7a951aff13379f15683f014cb449f3c711f14c25f19c7828b3d6dc1e"}
Sep 10 21:20:25.215976 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:25.215943    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n" event={"ID":"5baa9611-8f58-4257-8bdb-7ae8480003bc","Type":"ContainerStarted","Data":"5521aa19597e8694649501445988752c08269a74ddad552c9aa21c6f5656c5a9"}
Sep 10 21:20:25.216445 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:25.216029    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"
Sep 10 21:20:25.217661 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:25.217633    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf" event={"ID":"34ee329b-33c0-47a6-a8b6-442b9d266252","Type":"ContainerStarted","Data":"f2c647fca15399739d030216d51f9bc900d50c9e854d62d55c3ab2c90e0b90cb"}
Sep 10 21:20:25.217774 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:25.217692    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:20:25.233406 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:25.233361    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n" podStartSLOduration=2.23334606 podStartE2EDuration="2.23334606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:25.232087908 +0000 UTC m=+1366.207747958" watchObservedRunningTime="2026-09-10 21:20:25.23334606 +0000 UTC m=+1366.209006107"
Sep 10 21:20:25.251515 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:25.251465    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf" podStartSLOduration=2.730222678 podStartE2EDuration="3.251451991s" totalImagesPullingTime="521.229313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:25.249627466 +0000 UTC m=+1366.225287524" watchObservedRunningTime="2026-09-10 21:20:25.251451991 +0000 UTC m=+1366.227112093"
Sep 10 21:20:31.228434 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:31.228405    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:20:36.225116 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:36.225087    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"
Sep 10 21:20:51.108459 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.108419    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"]
Sep 10 21:20:51.112306 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.112285    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.114923 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.114903    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9f6475\"/\"kube-root-ca.crt\""
Sep 10 21:20:51.115734 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.115718    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9f6475\"/\"openshift-service-ca.crt\""
Sep 10 21:20:51.116061 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.116040    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9f6475\"/\"default-dockercfg-m4fdt\""
Sep 10 21:20:51.116165 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.116080    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9f6475\"/\"rate-test-model-9f6475-kserve-self-signed-certs\""
Sep 10 21:20:51.123159 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.123136    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"]
Sep 10 21:20:51.290635 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.290602    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf351188-3fec-40a2-aabb-cbdcd593f8f9-tls-certs\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.290808 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.290650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-model-cache\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.290808 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.290682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-home\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.290808 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.290785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-dshm\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.290929 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.290809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-tmp-dir\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.290929 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.290838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w485\" (UniqueName: \"kubernetes.io/projected/cf351188-3fec-40a2-aabb-cbdcd593f8f9-kube-api-access-2w485\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.391738 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.391659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-dshm\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.391738 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.391691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-tmp-dir\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.391738 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.391715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w485\" (UniqueName: \"kubernetes.io/projected/cf351188-3fec-40a2-aabb-cbdcd593f8f9-kube-api-access-2w485\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.391738 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.391739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf351188-3fec-40a2-aabb-cbdcd593f8f9-tls-certs\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.392007 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.391769    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-model-cache\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.392007 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.391789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-home\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.392184 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.392161    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-tmp-dir\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.392274 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.392177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-model-cache\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.392274 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.392242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-home\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.393922 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.393896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-dshm\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.394213 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.394189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf351188-3fec-40a2-aabb-cbdcd593f8f9-tls-certs\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.401339 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.401318    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w485\" (UniqueName: \"kubernetes.io/projected/cf351188-3fec-40a2-aabb-cbdcd593f8f9-kube-api-access-2w485\") pod \"rate-test-model-9f6475-kserve-7d5f5649c9-qvz68\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") " pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.423368 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.423343    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:20:51.696944 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:51.696922    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"]
Sep 10 21:20:51.699172 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:20:51.699146    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf351188_3fec_40a2_aabb_cbdcd593f8f9.slice/crio-965635d52cc0cae3fc1fcf64b565ca542317a334fc6235853a346fa941ca4577 WatchSource:0}: Error finding container 965635d52cc0cae3fc1fcf64b565ca542317a334fc6235853a346fa941ca4577: Status 404 returned error can't find the container with id 965635d52cc0cae3fc1fcf64b565ca542317a334fc6235853a346fa941ca4577
Sep 10 21:20:52.356440 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:52.356403    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68" event={"ID":"cf351188-3fec-40a2-aabb-cbdcd593f8f9","Type":"ContainerStarted","Data":"493b996d8c4e7a289e7e18ca2220e32cd6810dbb2563de1901f13cf7854f46d9"}
Sep 10 21:20:52.356440 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:52.356448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68" event={"ID":"cf351188-3fec-40a2-aabb-cbdcd593f8f9","Type":"ContainerStarted","Data":"965635d52cc0cae3fc1fcf64b565ca542317a334fc6235853a346fa941ca4577"}
Sep 10 21:20:52.373873 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:20:52.373821    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68" podStartSLOduration=1.373802936 podStartE2EDuration="1.373802936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:52.373225823 +0000 UTC m=+1393.348885861" watchObservedRunningTime="2026-09-10 21:20:52.373802936 +0000 UTC m=+1393.349462984"
Sep 10 21:21:01.423900 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:01.423866    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:21:01.424422 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:01.423917    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:21:01.436318 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:01.436297    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:21:02.419552 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:02.419523    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:21:05.400380 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.400346    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"]
Sep 10 21:21:05.400856 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.400574    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58" podUID="cde04da8-fedf-409c-adfc-fdd286e665d5" containerName="maas-api" containerID="cri-o://a7ee49893981b3919bfea01937e8df1071175e830ac96c26f9deaf9272ea958e" gracePeriod=30
Sep 10 21:21:05.457415 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.457379    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"]
Sep 10 21:21:05.457701 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.457650    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c" podUID="9dc7d43e-143b-46ad-bfcc-1a421c67a67d" containerName="payload-processing" containerID="cri-o://647070bf35a17069880bcb481c16b13b1a5d340934f5bf64b36cb25d3a37fdfa" gracePeriod=30
Sep 10 21:21:05.708379 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.708355    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:21:05.737755 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.737727    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"
Sep 10 21:21:05.826149 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.826124    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde04da8-fedf-409c-adfc-fdd286e665d5-openshift-service-ca\") pod \"cde04da8-fedf-409c-adfc-fdd286e665d5\" (UID: \"cde04da8-fedf-409c-adfc-fdd286e665d5\") "
Sep 10 21:21:05.826306 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.826183    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cde04da8-fedf-409c-adfc-fdd286e665d5-maas-api-tls\") pod \"cde04da8-fedf-409c-adfc-fdd286e665d5\" (UID: \"cde04da8-fedf-409c-adfc-fdd286e665d5\") "
Sep 10 21:21:05.826306 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.826250    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jz75\" (UniqueName: \"kubernetes.io/projected/9dc7d43e-143b-46ad-bfcc-1a421c67a67d-kube-api-access-6jz75\") pod \"9dc7d43e-143b-46ad-bfcc-1a421c67a67d\" (UID: \"9dc7d43e-143b-46ad-bfcc-1a421c67a67d\") "
Sep 10 21:21:05.826306 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.826291    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc7d43e-143b-46ad-bfcc-1a421c67a67d-plugins-config-volume\") pod \"9dc7d43e-143b-46ad-bfcc-1a421c67a67d\" (UID: \"9dc7d43e-143b-46ad-bfcc-1a421c67a67d\") "
Sep 10 21:21:05.826469 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.826372    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf6pg\" (UniqueName: \"kubernetes.io/projected/cde04da8-fedf-409c-adfc-fdd286e665d5-kube-api-access-nf6pg\") pod \"cde04da8-fedf-409c-adfc-fdd286e665d5\" (UID: \"cde04da8-fedf-409c-adfc-fdd286e665d5\") "
Sep 10 21:21:05.826469 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.826402    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde04da8-fedf-409c-adfc-fdd286e665d5-metrics-certs\") pod \"cde04da8-fedf-409c-adfc-fdd286e665d5\" (UID: \"cde04da8-fedf-409c-adfc-fdd286e665d5\") "
Sep 10 21:21:05.826581 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.826459    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde04da8-fedf-409c-adfc-fdd286e665d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cde04da8-fedf-409c-adfc-fdd286e665d5" (UID: "cde04da8-fedf-409c-adfc-fdd286e665d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:05.826787 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.826754    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc7d43e-143b-46ad-bfcc-1a421c67a67d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9dc7d43e-143b-46ad-bfcc-1a421c67a67d" (UID: "9dc7d43e-143b-46ad-bfcc-1a421c67a67d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:05.826929 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.826792    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde04da8-fedf-409c-adfc-fdd286e665d5-openshift-service-ca\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:05.828363 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.828339    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde04da8-fedf-409c-adfc-fdd286e665d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cde04da8-fedf-409c-adfc-fdd286e665d5" (UID: "cde04da8-fedf-409c-adfc-fdd286e665d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:05.828521 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.828371    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc7d43e-143b-46ad-bfcc-1a421c67a67d-kube-api-access-6jz75" (OuterVolumeSpecName: "kube-api-access-6jz75") pod "9dc7d43e-143b-46ad-bfcc-1a421c67a67d" (UID: "9dc7d43e-143b-46ad-bfcc-1a421c67a67d"). InnerVolumeSpecName "kube-api-access-6jz75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:05.828521 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.828426    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde04da8-fedf-409c-adfc-fdd286e665d5-kube-api-access-nf6pg" (OuterVolumeSpecName: "kube-api-access-nf6pg") pod "cde04da8-fedf-409c-adfc-fdd286e665d5" (UID: "cde04da8-fedf-409c-adfc-fdd286e665d5"). InnerVolumeSpecName "kube-api-access-nf6pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:05.828521 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.828435    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde04da8-fedf-409c-adfc-fdd286e665d5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cde04da8-fedf-409c-adfc-fdd286e665d5" (UID: "cde04da8-fedf-409c-adfc-fdd286e665d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:05.927920 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.927817    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc7d43e-143b-46ad-bfcc-1a421c67a67d-plugins-config-volume\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:05.927920 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.927859    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf6pg\" (UniqueName: \"kubernetes.io/projected/cde04da8-fedf-409c-adfc-fdd286e665d5-kube-api-access-nf6pg\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:05.927920 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.927872    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde04da8-fedf-409c-adfc-fdd286e665d5-metrics-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:05.927920 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.927883    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cde04da8-fedf-409c-adfc-fdd286e665d5-maas-api-tls\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:05.927920 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:05.927895    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jz75\" (UniqueName: \"kubernetes.io/projected/9dc7d43e-143b-46ad-bfcc-1a421c67a67d-kube-api-access-6jz75\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:06.430953 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.430919    2581 generic.go:356] "Generic (PLEG): container finished" podID="9dc7d43e-143b-46ad-bfcc-1a421c67a67d" containerID="647070bf35a17069880bcb481c16b13b1a5d340934f5bf64b36cb25d3a37fdfa" exitCode=0
Sep 10 21:21:06.431374 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.430969    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"
Sep 10 21:21:06.431374 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.431003    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c" event={"ID":"9dc7d43e-143b-46ad-bfcc-1a421c67a67d","Type":"ContainerDied","Data":"647070bf35a17069880bcb481c16b13b1a5d340934f5bf64b36cb25d3a37fdfa"}
Sep 10 21:21:06.431374 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.431058    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c" event={"ID":"9dc7d43e-143b-46ad-bfcc-1a421c67a67d","Type":"ContainerDied","Data":"a52020f84ed32b3e68eb5d2bf686330bfb826a818a0727764c1aa8ea690a084a"}
Sep 10 21:21:06.431374 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.431081    2581 scope.go:117] "RemoveContainer" containerID="647070bf35a17069880bcb481c16b13b1a5d340934f5bf64b36cb25d3a37fdfa"
Sep 10 21:21:06.432554 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.432514    2581 generic.go:356] "Generic (PLEG): container finished" podID="cde04da8-fedf-409c-adfc-fdd286e665d5" containerID="a7ee49893981b3919bfea01937e8df1071175e830ac96c26f9deaf9272ea958e" exitCode=0
Sep 10 21:21:06.432554 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.432550    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"
Sep 10 21:21:06.432696 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.432592    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58" event={"ID":"cde04da8-fedf-409c-adfc-fdd286e665d5","Type":"ContainerDied","Data":"a7ee49893981b3919bfea01937e8df1071175e830ac96c26f9deaf9272ea958e"}
Sep 10 21:21:06.432696 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.432642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58" event={"ID":"cde04da8-fedf-409c-adfc-fdd286e665d5","Type":"ContainerDied","Data":"b06e23dcd2be2b4435ac0a66f5f17f877263db91e66252e996f3ae9f0489867a"}
Sep 10 21:21:06.440062 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.440044    2581 scope.go:117] "RemoveContainer" containerID="647070bf35a17069880bcb481c16b13b1a5d340934f5bf64b36cb25d3a37fdfa"
Sep 10 21:21:06.440389 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:21:06.440363    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"647070bf35a17069880bcb481c16b13b1a5d340934f5bf64b36cb25d3a37fdfa\": container with ID starting with 647070bf35a17069880bcb481c16b13b1a5d340934f5bf64b36cb25d3a37fdfa not found: ID does not exist" containerID="647070bf35a17069880bcb481c16b13b1a5d340934f5bf64b36cb25d3a37fdfa"
Sep 10 21:21:06.440499 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.440397    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"647070bf35a17069880bcb481c16b13b1a5d340934f5bf64b36cb25d3a37fdfa"} err="failed to get container status \"647070bf35a17069880bcb481c16b13b1a5d340934f5bf64b36cb25d3a37fdfa\": rpc error: code = NotFound desc = could not find container \"647070bf35a17069880bcb481c16b13b1a5d340934f5bf64b36cb25d3a37fdfa\": container with ID starting with 647070bf35a17069880bcb481c16b13b1a5d340934f5bf64b36cb25d3a37fdfa not found: ID does not exist"
Sep 10 21:21:06.440499 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.440422    2581 scope.go:117] "RemoveContainer" containerID="a7ee49893981b3919bfea01937e8df1071175e830ac96c26f9deaf9272ea958e"
Sep 10 21:21:06.448718 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.448697    2581 scope.go:117] "RemoveContainer" containerID="a7ee49893981b3919bfea01937e8df1071175e830ac96c26f9deaf9272ea958e"
Sep 10 21:21:06.448973 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:21:06.448956    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ee49893981b3919bfea01937e8df1071175e830ac96c26f9deaf9272ea958e\": container with ID starting with a7ee49893981b3919bfea01937e8df1071175e830ac96c26f9deaf9272ea958e not found: ID does not exist" containerID="a7ee49893981b3919bfea01937e8df1071175e830ac96c26f9deaf9272ea958e"
Sep 10 21:21:06.449024 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.448981    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ee49893981b3919bfea01937e8df1071175e830ac96c26f9deaf9272ea958e"} err="failed to get container status \"a7ee49893981b3919bfea01937e8df1071175e830ac96c26f9deaf9272ea958e\": rpc error: code = NotFound desc = could not find container \"a7ee49893981b3919bfea01937e8df1071175e830ac96c26f9deaf9272ea958e\": container with ID starting with a7ee49893981b3919bfea01937e8df1071175e830ac96c26f9deaf9272ea958e not found: ID does not exist"
Sep 10 21:21:06.454926 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.454906    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"]
Sep 10 21:21:06.461429 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.461406    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0778a4-84c94f6fd5-q692c"]
Sep 10 21:21:06.471484 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.471465    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"]
Sep 10 21:21:06.475383 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:06.475364    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0778a4-567775cb9c-twd58"]
Sep 10 21:21:07.574623 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:07.574590    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dc7d43e-143b-46ad-bfcc-1a421c67a67d" path="/var/lib/kubelet/pods/9dc7d43e-143b-46ad-bfcc-1a421c67a67d/volumes"
Sep 10 21:21:07.574991 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:07.574977    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde04da8-fedf-409c-adfc-fdd286e665d5" path="/var/lib/kubelet/pods/cde04da8-fedf-409c-adfc-fdd286e665d5/volumes"
Sep 10 21:21:28.686257 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:28.686225    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"]
Sep 10 21:21:28.686763 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:28.686438    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf" podUID="34ee329b-33c0-47a6-a8b6-442b9d266252" containerName="maas-api" containerID="cri-o://f2c647fca15399739d030216d51f9bc900d50c9e854d62d55c3ab2c90e0b90cb" gracePeriod=30
Sep 10 21:21:28.743880 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:28.743845    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"]
Sep 10 21:21:28.744069 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:28.744044    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n" podUID="5baa9611-8f58-4257-8bdb-7ae8480003bc" containerName="payload-pre-processing" containerID="cri-o://5521aa19597e8694649501445988752c08269a74ddad552c9aa21c6f5656c5a9" gracePeriod=30
Sep 10 21:21:28.990939 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:28.990910    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:21:29.020541 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.020519    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"
Sep 10 21:21:29.155077 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.155046    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vcds\" (UniqueName: \"kubernetes.io/projected/34ee329b-33c0-47a6-a8b6-442b9d266252-kube-api-access-7vcds\") pod \"34ee329b-33c0-47a6-a8b6-442b9d266252\" (UID: \"34ee329b-33c0-47a6-a8b6-442b9d266252\") "
Sep 10 21:21:29.155290 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.155089    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5baa9611-8f58-4257-8bdb-7ae8480003bc-plugins-config-volume\") pod \"5baa9611-8f58-4257-8bdb-7ae8480003bc\" (UID: \"5baa9611-8f58-4257-8bdb-7ae8480003bc\") "
Sep 10 21:21:29.155290 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.155109    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtr4w\" (UniqueName: \"kubernetes.io/projected/5baa9611-8f58-4257-8bdb-7ae8480003bc-kube-api-access-gtr4w\") pod \"5baa9611-8f58-4257-8bdb-7ae8480003bc\" (UID: \"5baa9611-8f58-4257-8bdb-7ae8480003bc\") "
Sep 10 21:21:29.155290 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.155133    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ee329b-33c0-47a6-a8b6-442b9d266252-metrics-certs\") pod \"34ee329b-33c0-47a6-a8b6-442b9d266252\" (UID: \"34ee329b-33c0-47a6-a8b6-442b9d266252\") "
Sep 10 21:21:29.155290 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.155273    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ee329b-33c0-47a6-a8b6-442b9d266252-openshift-service-ca\") pod \"34ee329b-33c0-47a6-a8b6-442b9d266252\" (UID: \"34ee329b-33c0-47a6-a8b6-442b9d266252\") "
Sep 10 21:21:29.155530 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.155303    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ee329b-33c0-47a6-a8b6-442b9d266252-maas-api-tls\") pod \"34ee329b-33c0-47a6-a8b6-442b9d266252\" (UID: \"34ee329b-33c0-47a6-a8b6-442b9d266252\") "
Sep 10 21:21:29.155631 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.155476    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5baa9611-8f58-4257-8bdb-7ae8480003bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5baa9611-8f58-4257-8bdb-7ae8480003bc" (UID: "5baa9611-8f58-4257-8bdb-7ae8480003bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:29.155691 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.155655    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ee329b-33c0-47a6-a8b6-442b9d266252-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "34ee329b-33c0-47a6-a8b6-442b9d266252" (UID: "34ee329b-33c0-47a6-a8b6-442b9d266252"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:29.157428 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.157401    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5baa9611-8f58-4257-8bdb-7ae8480003bc-kube-api-access-gtr4w" (OuterVolumeSpecName: "kube-api-access-gtr4w") pod "5baa9611-8f58-4257-8bdb-7ae8480003bc" (UID: "5baa9611-8f58-4257-8bdb-7ae8480003bc"). InnerVolumeSpecName "kube-api-access-gtr4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:29.157541 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.157444    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ee329b-33c0-47a6-a8b6-442b9d266252-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "34ee329b-33c0-47a6-a8b6-442b9d266252" (UID: "34ee329b-33c0-47a6-a8b6-442b9d266252"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:29.157694 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.157672    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ee329b-33c0-47a6-a8b6-442b9d266252-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "34ee329b-33c0-47a6-a8b6-442b9d266252" (UID: "34ee329b-33c0-47a6-a8b6-442b9d266252"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:29.157694 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.157684    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ee329b-33c0-47a6-a8b6-442b9d266252-kube-api-access-7vcds" (OuterVolumeSpecName: "kube-api-access-7vcds") pod "34ee329b-33c0-47a6-a8b6-442b9d266252" (UID: "34ee329b-33c0-47a6-a8b6-442b9d266252"). InnerVolumeSpecName "kube-api-access-7vcds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:29.256708 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.256634    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vcds\" (UniqueName: \"kubernetes.io/projected/34ee329b-33c0-47a6-a8b6-442b9d266252-kube-api-access-7vcds\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:29.256708 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.256660    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5baa9611-8f58-4257-8bdb-7ae8480003bc-plugins-config-volume\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:29.256708 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.256670    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtr4w\" (UniqueName: \"kubernetes.io/projected/5baa9611-8f58-4257-8bdb-7ae8480003bc-kube-api-access-gtr4w\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:29.256708 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.256679    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ee329b-33c0-47a6-a8b6-442b9d266252-metrics-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:29.256708 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.256688    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ee329b-33c0-47a6-a8b6-442b9d266252-openshift-service-ca\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:29.256708 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.256697    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ee329b-33c0-47a6-a8b6-442b9d266252-maas-api-tls\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:29.554645 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.554614    2581 generic.go:356] "Generic (PLEG): container finished" podID="5baa9611-8f58-4257-8bdb-7ae8480003bc" containerID="5521aa19597e8694649501445988752c08269a74ddad552c9aa21c6f5656c5a9" exitCode=0
Sep 10 21:21:29.554826 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.554658    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"
Sep 10 21:21:29.554826 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.554691    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n" event={"ID":"5baa9611-8f58-4257-8bdb-7ae8480003bc","Type":"ContainerDied","Data":"5521aa19597e8694649501445988752c08269a74ddad552c9aa21c6f5656c5a9"}
Sep 10 21:21:29.554826 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.554755    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n" event={"ID":"5baa9611-8f58-4257-8bdb-7ae8480003bc","Type":"ContainerDied","Data":"cd8d768144196aec81a46eea32c538516964a5ca75394e661823f4914958a222"}
Sep 10 21:21:29.554826 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.554783    2581 scope.go:117] "RemoveContainer" containerID="5521aa19597e8694649501445988752c08269a74ddad552c9aa21c6f5656c5a9"
Sep 10 21:21:29.555986 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.555962    2581 generic.go:356] "Generic (PLEG): container finished" podID="34ee329b-33c0-47a6-a8b6-442b9d266252" containerID="f2c647fca15399739d030216d51f9bc900d50c9e854d62d55c3ab2c90e0b90cb" exitCode=0
Sep 10 21:21:29.556098 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.556031    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"
Sep 10 21:21:29.556098 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.556050    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf" event={"ID":"34ee329b-33c0-47a6-a8b6-442b9d266252","Type":"ContainerDied","Data":"f2c647fca15399739d030216d51f9bc900d50c9e854d62d55c3ab2c90e0b90cb"}
Sep 10 21:21:29.556175 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.556114    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf" event={"ID":"34ee329b-33c0-47a6-a8b6-442b9d266252","Type":"ContainerDied","Data":"61bd5afc7a951aff13379f15683f014cb449f3c711f14c25f19c7828b3d6dc1e"}
Sep 10 21:21:29.564879 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.564860    2581 scope.go:117] "RemoveContainer" containerID="5521aa19597e8694649501445988752c08269a74ddad552c9aa21c6f5656c5a9"
Sep 10 21:21:29.565163 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:21:29.565140    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5521aa19597e8694649501445988752c08269a74ddad552c9aa21c6f5656c5a9\": container with ID starting with 5521aa19597e8694649501445988752c08269a74ddad552c9aa21c6f5656c5a9 not found: ID does not exist" containerID="5521aa19597e8694649501445988752c08269a74ddad552c9aa21c6f5656c5a9"
Sep 10 21:21:29.565230 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.565176    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5521aa19597e8694649501445988752c08269a74ddad552c9aa21c6f5656c5a9"} err="failed to get container status \"5521aa19597e8694649501445988752c08269a74ddad552c9aa21c6f5656c5a9\": rpc error: code = NotFound desc = could not find container \"5521aa19597e8694649501445988752c08269a74ddad552c9aa21c6f5656c5a9\": container with ID starting with 5521aa19597e8694649501445988752c08269a74ddad552c9aa21c6f5656c5a9 not found: ID does not exist"
Sep 10 21:21:29.565296 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.565238    2581 scope.go:117] "RemoveContainer" containerID="f2c647fca15399739d030216d51f9bc900d50c9e854d62d55c3ab2c90e0b90cb"
Sep 10 21:21:29.580596 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.580575    2581 scope.go:117] "RemoveContainer" containerID="f2c647fca15399739d030216d51f9bc900d50c9e854d62d55c3ab2c90e0b90cb"
Sep 10 21:21:29.580894 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:21:29.580872    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c647fca15399739d030216d51f9bc900d50c9e854d62d55c3ab2c90e0b90cb\": container with ID starting with f2c647fca15399739d030216d51f9bc900d50c9e854d62d55c3ab2c90e0b90cb not found: ID does not exist" containerID="f2c647fca15399739d030216d51f9bc900d50c9e854d62d55c3ab2c90e0b90cb"
Sep 10 21:21:29.580983 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.580899    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c647fca15399739d030216d51f9bc900d50c9e854d62d55c3ab2c90e0b90cb"} err="failed to get container status \"f2c647fca15399739d030216d51f9bc900d50c9e854d62d55c3ab2c90e0b90cb\": rpc error: code = NotFound desc = could not find container \"f2c647fca15399739d030216d51f9bc900d50c9e854d62d55c3ab2c90e0b90cb\": container with ID starting with f2c647fca15399739d030216d51f9bc900d50c9e854d62d55c3ab2c90e0b90cb not found: ID does not exist"
Sep 10 21:21:29.582843 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.582823    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"]
Sep 10 21:21:29.582927 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.582850    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-806a33-5cd6f66798-sj66n"]
Sep 10 21:21:29.601453 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.601429    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"]
Sep 10 21:21:29.611458 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:29.611436    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-806a33-6545679d48-rfbtf"]
Sep 10 21:21:31.482277 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:31.482243    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"]
Sep 10 21:21:31.573586 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:31.573557    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34ee329b-33c0-47a6-a8b6-442b9d266252" path="/var/lib/kubelet/pods/34ee329b-33c0-47a6-a8b6-442b9d266252/volumes"
Sep 10 21:21:31.573914 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:31.573902    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5baa9611-8f58-4257-8bdb-7ae8480003bc" path="/var/lib/kubelet/pods/5baa9611-8f58-4257-8bdb-7ae8480003bc/volumes"
Sep 10 21:21:37.310048 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.309956    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"]
Sep 10 21:21:37.310595 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.310576    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc7d43e-143b-46ad-bfcc-1a421c67a67d" containerName="payload-processing"
Sep 10 21:21:37.310674 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.310596    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc7d43e-143b-46ad-bfcc-1a421c67a67d" containerName="payload-processing"
Sep 10 21:21:37.310674 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.310610    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34ee329b-33c0-47a6-a8b6-442b9d266252" containerName="maas-api"
Sep 10 21:21:37.310674 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.310619    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="34ee329b-33c0-47a6-a8b6-442b9d266252" containerName="maas-api"
Sep 10 21:21:37.310674 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.310635    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5baa9611-8f58-4257-8bdb-7ae8480003bc" containerName="payload-pre-processing"
Sep 10 21:21:37.310674 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.310645    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5baa9611-8f58-4257-8bdb-7ae8480003bc" containerName="payload-pre-processing"
Sep 10 21:21:37.310940 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.310689    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde04da8-fedf-409c-adfc-fdd286e665d5" containerName="maas-api"
Sep 10 21:21:37.310940 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.310698    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde04da8-fedf-409c-adfc-fdd286e665d5" containerName="maas-api"
Sep 10 21:21:37.310940 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.310780    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5baa9611-8f58-4257-8bdb-7ae8480003bc" containerName="payload-pre-processing"
Sep 10 21:21:37.310940 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.310791    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dc7d43e-143b-46ad-bfcc-1a421c67a67d" containerName="payload-processing"
Sep 10 21:21:37.310940 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.310802    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde04da8-fedf-409c-adfc-fdd286e665d5" containerName="maas-api"
Sep 10 21:21:37.310940 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.310814    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="34ee329b-33c0-47a6-a8b6-442b9d266252" containerName="maas-api"
Sep 10 21:21:37.315727 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.315700    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:21:37.317900 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.317854    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f7eb29-serving-cert\""
Sep 10 21:21:37.321947 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.321873    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"]
Sep 10 21:21:37.427509 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.427479    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58de574f-05c8-4efc-b4aa-b8de6d07fd41-metrics-certs\") pod \"maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd\" (UID: \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:21:37.427672 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.427516    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58de574f-05c8-4efc-b4aa-b8de6d07fd41-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd\" (UID: \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:21:37.427672 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.427538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58de574f-05c8-4efc-b4aa-b8de6d07fd41-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd\" (UID: \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:21:37.427672 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.427653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckj29\" (UniqueName: \"kubernetes.io/projected/58de574f-05c8-4efc-b4aa-b8de6d07fd41-kube-api-access-ckj29\") pod \"maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd\" (UID: \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:21:37.528645 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.528611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58de574f-05c8-4efc-b4aa-b8de6d07fd41-metrics-certs\") pod \"maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd\" (UID: \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:21:37.528827 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.528654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58de574f-05c8-4efc-b4aa-b8de6d07fd41-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd\" (UID: \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:21:37.528827 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.528675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58de574f-05c8-4efc-b4aa-b8de6d07fd41-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd\" (UID: \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:21:37.528827 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.528715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckj29\" (UniqueName: \"kubernetes.io/projected/58de574f-05c8-4efc-b4aa-b8de6d07fd41-kube-api-access-ckj29\") pod \"maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd\" (UID: \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:21:37.529355 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.529318    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58de574f-05c8-4efc-b4aa-b8de6d07fd41-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd\" (UID: \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:21:37.531059 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.531038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58de574f-05c8-4efc-b4aa-b8de6d07fd41-metrics-certs\") pod \"maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd\" (UID: \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:21:37.531157 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.531107    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58de574f-05c8-4efc-b4aa-b8de6d07fd41-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd\" (UID: \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:21:37.536881 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.536854    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckj29\" (UniqueName: \"kubernetes.io/projected/58de574f-05c8-4efc-b4aa-b8de6d07fd41-kube-api-access-ckj29\") pod \"maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd\" (UID: \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:21:37.628523 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.628449    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:21:37.905658 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:37.905634    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"]
Sep 10 21:21:37.907994 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:21:37.907954    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58de574f_05c8_4efc_b4aa_b8de6d07fd41.slice/crio-b6d686bafebd16738371e9b6b5fe9c3532b6fd27a1d07440948f14255b86348d WatchSource:0}: Error finding container b6d686bafebd16738371e9b6b5fe9c3532b6fd27a1d07440948f14255b86348d: Status 404 returned error can't find the container with id b6d686bafebd16738371e9b6b5fe9c3532b6fd27a1d07440948f14255b86348d
Sep 10 21:21:38.162141 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:38.162066    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"]
Sep 10 21:21:38.166798 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:38.166781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"
Sep 10 21:21:38.169000 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:38.168978    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f7eb29-dockercfg-6wg4h\""
Sep 10 21:21:38.169119 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:38.168977    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f7eb29\""
Sep 10 21:21:38.174404 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:38.174367    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"]
Sep 10 21:21:38.235996 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:38.235972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgm9m\" (UniqueName: \"kubernetes.io/projected/b31d8328-f34d-4d6e-b284-8b6261ec2971-kube-api-access-sgm9m\") pod \"payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst\" (UID: \"b31d8328-f34d-4d6e-b284-8b6261ec2971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"
Sep 10 21:21:38.236150 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:38.236031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31d8328-f34d-4d6e-b284-8b6261ec2971-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst\" (UID: \"b31d8328-f34d-4d6e-b284-8b6261ec2971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"
Sep 10 21:21:38.337228 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:38.337192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgm9m\" (UniqueName: \"kubernetes.io/projected/b31d8328-f34d-4d6e-b284-8b6261ec2971-kube-api-access-sgm9m\") pod \"payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst\" (UID: \"b31d8328-f34d-4d6e-b284-8b6261ec2971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"
Sep 10 21:21:38.337589 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:38.337263    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31d8328-f34d-4d6e-b284-8b6261ec2971-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst\" (UID: \"b31d8328-f34d-4d6e-b284-8b6261ec2971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"
Sep 10 21:21:38.338109 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:38.338089    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31d8328-f34d-4d6e-b284-8b6261ec2971-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst\" (UID: \"b31d8328-f34d-4d6e-b284-8b6261ec2971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"
Sep 10 21:21:38.346489 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:38.346470    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgm9m\" (UniqueName: \"kubernetes.io/projected/b31d8328-f34d-4d6e-b284-8b6261ec2971-kube-api-access-sgm9m\") pod \"payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst\" (UID: \"b31d8328-f34d-4d6e-b284-8b6261ec2971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"
Sep 10 21:21:38.478122 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:38.478040    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"
Sep 10 21:21:38.604062 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:38.604028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd" event={"ID":"58de574f-05c8-4efc-b4aa-b8de6d07fd41","Type":"ContainerStarted","Data":"ea7300f853b29f826f1c74fb0846c8536d52718d2e72d5a74c036cca79e76caa"}
Sep 10 21:21:38.604237 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:38.604078    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd" event={"ID":"58de574f-05c8-4efc-b4aa-b8de6d07fd41","Type":"ContainerStarted","Data":"b6d686bafebd16738371e9b6b5fe9c3532b6fd27a1d07440948f14255b86348d"}
Sep 10 21:21:38.604237 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:38.604180    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:21:38.622128 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:38.622086    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd" podStartSLOduration=1.228230111 podStartE2EDuration="1.622072742s" totalImagesPullingTime="393.842631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:21:38.619783066 +0000 UTC m=+1439.595443113" watchObservedRunningTime="2026-09-10 21:21:38.622072742 +0000 UTC m=+1439.597732788"
Sep 10 21:21:38.751144 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:38.751118    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"]
Sep 10 21:21:38.753225 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:21:38.753179    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31d8328_f34d_4d6e_b284_8b6261ec2971.slice/crio-b6eb1dd23c41579a60c6182d8b25e3528df730a60bf829ad9024708b4369cab5 WatchSource:0}: Error finding container b6eb1dd23c41579a60c6182d8b25e3528df730a60bf829ad9024708b4369cab5: Status 404 returned error can't find the container with id b6eb1dd23c41579a60c6182d8b25e3528df730a60bf829ad9024708b4369cab5
Sep 10 21:21:39.610961 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:39.610924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst" event={"ID":"b31d8328-f34d-4d6e-b284-8b6261ec2971","Type":"ContainerStarted","Data":"aaaae8b983fe5d252c3e76058a4cff092fb51b107e221cdb835f833cb0e2ac6d"}
Sep 10 21:21:39.611318 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:39.610993    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"
Sep 10 21:21:39.611318 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:39.611017    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst" event={"ID":"b31d8328-f34d-4d6e-b284-8b6261ec2971","Type":"ContainerStarted","Data":"b6eb1dd23c41579a60c6182d8b25e3528df730a60bf829ad9024708b4369cab5"}
Sep 10 21:21:39.629435 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:39.629389    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst" podStartSLOduration=1.629373958 podStartE2EDuration="1.629373958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:21:39.626800619 +0000 UTC m=+1440.602460666" watchObservedRunningTime="2026-09-10 21:21:39.629373958 +0000 UTC m=+1440.605034004"
Sep 10 21:21:44.616355 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:44.616324    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:21:46.501942 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.501904    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68" podUID="cf351188-3fec-40a2-aabb-cbdcd593f8f9" containerName="main" containerID="cri-o://493b996d8c4e7a289e7e18ca2220e32cd6810dbb2563de1901f13cf7854f46d9" gracePeriod=45
Sep 10 21:21:46.648302 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.648268    2581 generic.go:356] "Generic (PLEG): container finished" podID="cf351188-3fec-40a2-aabb-cbdcd593f8f9" containerID="493b996d8c4e7a289e7e18ca2220e32cd6810dbb2563de1901f13cf7854f46d9" exitCode=0
Sep 10 21:21:46.648461 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.648334    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68" event={"ID":"cf351188-3fec-40a2-aabb-cbdcd593f8f9","Type":"ContainerDied","Data":"493b996d8c4e7a289e7e18ca2220e32cd6810dbb2563de1901f13cf7854f46d9"}
Sep 10 21:21:46.755138 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.755078    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:21:46.813362 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.813329    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-tmp-dir\") pod \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") "
Sep 10 21:21:46.813502 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.813393    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf351188-3fec-40a2-aabb-cbdcd593f8f9-tls-certs\") pod \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") "
Sep 10 21:21:46.813502 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.813437    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w485\" (UniqueName: \"kubernetes.io/projected/cf351188-3fec-40a2-aabb-cbdcd593f8f9-kube-api-access-2w485\") pod \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") "
Sep 10 21:21:46.813589 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.813549    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-dshm\") pod \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") "
Sep 10 21:21:46.813589 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.813576    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-home\") pod \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") "
Sep 10 21:21:46.813699 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.813603    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf351188-3fec-40a2-aabb-cbdcd593f8f9" (UID: "cf351188-3fec-40a2-aabb-cbdcd593f8f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:46.813699 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.813640    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-model-cache\") pod \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\" (UID: \"cf351188-3fec-40a2-aabb-cbdcd593f8f9\") "
Sep 10 21:21:46.813861 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.813838    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-home" (OuterVolumeSpecName: "home") pod "cf351188-3fec-40a2-aabb-cbdcd593f8f9" (UID: "cf351188-3fec-40a2-aabb-cbdcd593f8f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:46.813926 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.813901    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf351188-3fec-40a2-aabb-cbdcd593f8f9" (UID: "cf351188-3fec-40a2-aabb-cbdcd593f8f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:46.813970 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.813944    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-home\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:46.814002 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.813967    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-tmp-dir\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:46.815545 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.815515    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-dshm" (OuterVolumeSpecName: "dshm") pod "cf351188-3fec-40a2-aabb-cbdcd593f8f9" (UID: "cf351188-3fec-40a2-aabb-cbdcd593f8f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:46.815654 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.815543    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf351188-3fec-40a2-aabb-cbdcd593f8f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf351188-3fec-40a2-aabb-cbdcd593f8f9" (UID: "cf351188-3fec-40a2-aabb-cbdcd593f8f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:46.815654 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.815557    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf351188-3fec-40a2-aabb-cbdcd593f8f9-kube-api-access-2w485" (OuterVolumeSpecName: "kube-api-access-2w485") pod "cf351188-3fec-40a2-aabb-cbdcd593f8f9" (UID: "cf351188-3fec-40a2-aabb-cbdcd593f8f9"). InnerVolumeSpecName "kube-api-access-2w485". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:46.914485 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.914447    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf351188-3fec-40a2-aabb-cbdcd593f8f9-tls-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:46.914485 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.914479    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w485\" (UniqueName: \"kubernetes.io/projected/cf351188-3fec-40a2-aabb-cbdcd593f8f9-kube-api-access-2w485\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:46.914485 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.914490    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-dshm\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:46.914713 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:46.914499    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf351188-3fec-40a2-aabb-cbdcd593f8f9-model-cache\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:47.006361 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.006269    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"]
Sep 10 21:21:47.006542 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.006512    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk" podUID="2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36" containerName="maas-api" containerID="cri-o://414e138079689246a257d1f455d0b6da2eaccf5ced34e1bab1606e4a4b2a9e9a" gracePeriod=30
Sep 10 21:21:47.251642 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.251618    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:21:47.318149 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.318123    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-openshift-service-ca\") pod \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\" (UID: \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\") "
Sep 10 21:21:47.318344 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.318167    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-metrics-certs\") pod \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\" (UID: \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\") "
Sep 10 21:21:47.318344 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.318250    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmdt8\" (UniqueName: \"kubernetes.io/projected/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-kube-api-access-fmdt8\") pod \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\" (UID: \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\") "
Sep 10 21:21:47.318344 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.318270    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-maas-api-tls\") pod \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\" (UID: \"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36\") "
Sep 10 21:21:47.318553 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.318520    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36" (UID: "2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:47.318641 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.318623    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-openshift-service-ca\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:47.320348 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.320316    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36" (UID: "2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:47.320737 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.320718    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36" (UID: "2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:47.320796 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.320740    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-kube-api-access-fmdt8" (OuterVolumeSpecName: "kube-api-access-fmdt8") pod "2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36" (UID: "2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36"). InnerVolumeSpecName "kube-api-access-fmdt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:47.419196 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.419150    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-metrics-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:47.419196 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.419188    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmdt8\" (UniqueName: \"kubernetes.io/projected/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-kube-api-access-fmdt8\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:47.419196 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.419213    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36-maas-api-tls\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:21:47.655595 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.655553    2581 generic.go:356] "Generic (PLEG): container finished" podID="2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36" containerID="414e138079689246a257d1f455d0b6da2eaccf5ced34e1bab1606e4a4b2a9e9a" exitCode=0
Sep 10 21:21:47.656104 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.655603    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"
Sep 10 21:21:47.656104 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.655638    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk" event={"ID":"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36","Type":"ContainerDied","Data":"414e138079689246a257d1f455d0b6da2eaccf5ced34e1bab1606e4a4b2a9e9a"}
Sep 10 21:21:47.656104 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.655694    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk" event={"ID":"2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36","Type":"ContainerDied","Data":"d29d00a8b649b2143db70d92d1163ca85bdea480074f2decbcb630f477cd6bb6"}
Sep 10 21:21:47.656104 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.655727    2581 scope.go:117] "RemoveContainer" containerID="414e138079689246a257d1f455d0b6da2eaccf5ced34e1bab1606e4a4b2a9e9a"
Sep 10 21:21:47.657300 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.657283    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"
Sep 10 21:21:47.657382 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.657327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68" event={"ID":"cf351188-3fec-40a2-aabb-cbdcd593f8f9","Type":"ContainerDied","Data":"965635d52cc0cae3fc1fcf64b565ca542317a334fc6235853a346fa941ca4577"}
Sep 10 21:21:47.665007 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.664991    2581 scope.go:117] "RemoveContainer" containerID="414e138079689246a257d1f455d0b6da2eaccf5ced34e1bab1606e4a4b2a9e9a"
Sep 10 21:21:47.665313 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:21:47.665293    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414e138079689246a257d1f455d0b6da2eaccf5ced34e1bab1606e4a4b2a9e9a\": container with ID starting with 414e138079689246a257d1f455d0b6da2eaccf5ced34e1bab1606e4a4b2a9e9a not found: ID does not exist" containerID="414e138079689246a257d1f455d0b6da2eaccf5ced34e1bab1606e4a4b2a9e9a"
Sep 10 21:21:47.665379 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.665321    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414e138079689246a257d1f455d0b6da2eaccf5ced34e1bab1606e4a4b2a9e9a"} err="failed to get container status \"414e138079689246a257d1f455d0b6da2eaccf5ced34e1bab1606e4a4b2a9e9a\": rpc error: code = NotFound desc = could not find container \"414e138079689246a257d1f455d0b6da2eaccf5ced34e1bab1606e4a4b2a9e9a\": container with ID starting with 414e138079689246a257d1f455d0b6da2eaccf5ced34e1bab1606e4a4b2a9e9a not found: ID does not exist"
Sep 10 21:21:47.665379 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.665337    2581 scope.go:117] "RemoveContainer" containerID="493b996d8c4e7a289e7e18ca2220e32cd6810dbb2563de1901f13cf7854f46d9"
Sep 10 21:21:47.674755 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.674732    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"]
Sep 10 21:21:47.676607 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.676588    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-14cee3-5fc65fb7c8-4rrjk"]
Sep 10 21:21:47.690278 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.690248    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"]
Sep 10 21:21:47.695938 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:47.695920    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9f6475/rate-test-model-9f6475-kserve-7d5f5649c9-qvz68"]
Sep 10 21:21:49.574374 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:49.574341    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36" path="/var/lib/kubelet/pods/2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36/volumes"
Sep 10 21:21:49.574799 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:49.574779    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf351188-3fec-40a2-aabb-cbdcd593f8f9" path="/var/lib/kubelet/pods/cf351188-3fec-40a2-aabb-cbdcd593f8f9/volumes"
Sep 10 21:21:50.619087 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:21:50.619058    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"
Sep 10 21:22:04.835572 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:04.835541    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"]
Sep 10 21:22:04.836125 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:04.835721    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6" podUID="cce8decc-17da-409f-8af6-c94255909688" containerName="payload-pre-processing" containerID="cri-o://7fcc73cdf6ddcddb1e2b4e5e0a31ee63e3bc8dd52e32cb26295f3e26274de5fc" gracePeriod=30
Sep 10 21:22:05.084805 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:05.084782    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"
Sep 10 21:22:05.190070 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:05.189972    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce8decc-17da-409f-8af6-c94255909688-plugins-config-volume\") pod \"cce8decc-17da-409f-8af6-c94255909688\" (UID: \"cce8decc-17da-409f-8af6-c94255909688\") "
Sep 10 21:22:05.190275 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:05.190071    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvrs2\" (UniqueName: \"kubernetes.io/projected/cce8decc-17da-409f-8af6-c94255909688-kube-api-access-mvrs2\") pod \"cce8decc-17da-409f-8af6-c94255909688\" (UID: \"cce8decc-17da-409f-8af6-c94255909688\") "
Sep 10 21:22:05.190396 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:05.190373    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce8decc-17da-409f-8af6-c94255909688-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cce8decc-17da-409f-8af6-c94255909688" (UID: "cce8decc-17da-409f-8af6-c94255909688"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:22:05.192080 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:05.192059    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce8decc-17da-409f-8af6-c94255909688-kube-api-access-mvrs2" (OuterVolumeSpecName: "kube-api-access-mvrs2") pod "cce8decc-17da-409f-8af6-c94255909688" (UID: "cce8decc-17da-409f-8af6-c94255909688"). InnerVolumeSpecName "kube-api-access-mvrs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:22:05.291906 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:05.291873    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvrs2\" (UniqueName: \"kubernetes.io/projected/cce8decc-17da-409f-8af6-c94255909688-kube-api-access-mvrs2\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:22:05.291906 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:05.291899    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce8decc-17da-409f-8af6-c94255909688-plugins-config-volume\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:22:05.759070 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:05.759041    2581 generic.go:356] "Generic (PLEG): container finished" podID="cce8decc-17da-409f-8af6-c94255909688" containerID="7fcc73cdf6ddcddb1e2b4e5e0a31ee63e3bc8dd52e32cb26295f3e26274de5fc" exitCode=0
Sep 10 21:22:05.759260 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:05.759085    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"
Sep 10 21:22:05.759260 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:05.759137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6" event={"ID":"cce8decc-17da-409f-8af6-c94255909688","Type":"ContainerDied","Data":"7fcc73cdf6ddcddb1e2b4e5e0a31ee63e3bc8dd52e32cb26295f3e26274de5fc"}
Sep 10 21:22:05.759260 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:05.759192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6" event={"ID":"cce8decc-17da-409f-8af6-c94255909688","Type":"ContainerDied","Data":"83d5ad90aad1f2f08ae017c4ca0dc0de86e19f681d0944cda17d98790c4fb87a"}
Sep 10 21:22:05.759260 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:05.759236    2581 scope.go:117] "RemoveContainer" containerID="7fcc73cdf6ddcddb1e2b4e5e0a31ee63e3bc8dd52e32cb26295f3e26274de5fc"
Sep 10 21:22:05.767711 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:05.767691    2581 scope.go:117] "RemoveContainer" containerID="7fcc73cdf6ddcddb1e2b4e5e0a31ee63e3bc8dd52e32cb26295f3e26274de5fc"
Sep 10 21:22:05.767988 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:22:05.767969    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fcc73cdf6ddcddb1e2b4e5e0a31ee63e3bc8dd52e32cb26295f3e26274de5fc\": container with ID starting with 7fcc73cdf6ddcddb1e2b4e5e0a31ee63e3bc8dd52e32cb26295f3e26274de5fc not found: ID does not exist" containerID="7fcc73cdf6ddcddb1e2b4e5e0a31ee63e3bc8dd52e32cb26295f3e26274de5fc"
Sep 10 21:22:05.768036 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:05.767997    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fcc73cdf6ddcddb1e2b4e5e0a31ee63e3bc8dd52e32cb26295f3e26274de5fc"} err="failed to get container status \"7fcc73cdf6ddcddb1e2b4e5e0a31ee63e3bc8dd52e32cb26295f3e26274de5fc\": rpc error: code = NotFound desc = could not find container \"7fcc73cdf6ddcddb1e2b4e5e0a31ee63e3bc8dd52e32cb26295f3e26274de5fc\": container with ID starting with 7fcc73cdf6ddcddb1e2b4e5e0a31ee63e3bc8dd52e32cb26295f3e26274de5fc not found: ID does not exist"
Sep 10 21:22:05.777779 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:05.777759    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"]
Sep 10 21:22:05.782083 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:05.782063    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aca18f-55c8b5c6db-v6sq6"]
Sep 10 21:22:07.574769 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:07.574735    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cce8decc-17da-409f-8af6-c94255909688" path="/var/lib/kubelet/pods/cce8decc-17da-409f-8af6-c94255909688/volumes"
Sep 10 21:22:35.353998 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:35.353965    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kvhz2_ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f/payload-processing/0.log"
Sep 10 21:22:37.308068 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.308033    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"]
Sep 10 21:22:37.308589 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.308571    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce8decc-17da-409f-8af6-c94255909688" containerName="payload-pre-processing"
Sep 10 21:22:37.308589 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.308590    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce8decc-17da-409f-8af6-c94255909688" containerName="payload-pre-processing"
Sep 10 21:22:37.308719 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.308613    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36" containerName="maas-api"
Sep 10 21:22:37.308719 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.308623    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36" containerName="maas-api"
Sep 10 21:22:37.308719 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.308632    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf351188-3fec-40a2-aabb-cbdcd593f8f9" containerName="main"
Sep 10 21:22:37.308719 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.308639    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf351188-3fec-40a2-aabb-cbdcd593f8f9" containerName="main"
Sep 10 21:22:37.308719 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.308718    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cf0e4a7-594b-4706-af6b-2c2bf6cd2b36" containerName="maas-api"
Sep 10 21:22:37.308880 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.308726    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cce8decc-17da-409f-8af6-c94255909688" containerName="payload-pre-processing"
Sep 10 21:22:37.308880 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.308734    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf351188-3fec-40a2-aabb-cbdcd593f8f9" containerName="main"
Sep 10 21:22:37.311901 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.311883    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.315034 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.315013    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f7eb29\"/\"openshift-service-ca.crt\""
Sep 10 21:22:37.315149 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.315050    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f7eb29\"/\"ipp-route-f7eb29-kserve-self-signed-certs\""
Sep 10 21:22:37.315304 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.315289    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f7eb29\"/\"kube-root-ca.crt\""
Sep 10 21:22:37.315384 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.315322    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f7eb29\"/\"default-dockercfg-n4bjg\""
Sep 10 21:22:37.322316 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.322294    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"]
Sep 10 21:22:37.383067 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.383032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-tmp-dir\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.383257 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.383090    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-home\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.383257 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.383113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-dshm\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.383257 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.383132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-model-cache\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.383257 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.383175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw8md\" (UniqueName: \"kubernetes.io/projected/295308a1-23cc-4eac-8832-9c123ed28034-kube-api-access-mw8md\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.383257 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.383233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295308a1-23cc-4eac-8832-9c123ed28034-tls-certs\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.484362 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.484325    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw8md\" (UniqueName: \"kubernetes.io/projected/295308a1-23cc-4eac-8832-9c123ed28034-kube-api-access-mw8md\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.484362 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.484360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295308a1-23cc-4eac-8832-9c123ed28034-tls-certs\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.484577 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.484415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-tmp-dir\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.484577 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.484469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-home\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.484577 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.484495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-dshm\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.484577 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.484519    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-model-cache\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.484897 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.484873    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-tmp-dir\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.484977 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.484943    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-model-cache\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.484977 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.484938    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-home\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.486723 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.486701    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-dshm\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.486830 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.486815    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295308a1-23cc-4eac-8832-9c123ed28034-tls-certs\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.492842 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.492824    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw8md\" (UniqueName: \"kubernetes.io/projected/295308a1-23cc-4eac-8832-9c123ed28034-kube-api-access-mw8md\") pod \"ipp-route-f7eb29-kserve-674f894567-85vfg\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") " pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.622253 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.622145    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:37.897865 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.897783    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"]
Sep 10 21:22:37.900609 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:22:37.900583    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod295308a1_23cc_4eac_8832_9c123ed28034.slice/crio-c92c94b5c75ec46503df689ee14c116aa6809986ed2cfb23517c88b794cc6d43 WatchSource:0}: Error finding container c92c94b5c75ec46503df689ee14c116aa6809986ed2cfb23517c88b794cc6d43: Status 404 returned error can't find the container with id c92c94b5c75ec46503df689ee14c116aa6809986ed2cfb23517c88b794cc6d43
Sep 10 21:22:37.921461 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:37.921435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg" event={"ID":"295308a1-23cc-4eac-8832-9c123ed28034","Type":"ContainerStarted","Data":"c92c94b5c75ec46503df689ee14c116aa6809986ed2cfb23517c88b794cc6d43"}
Sep 10 21:22:38.928197 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:38.928162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg" event={"ID":"295308a1-23cc-4eac-8832-9c123ed28034","Type":"ContainerStarted","Data":"82aea308ee181d07ba5b48a2de8eee4a8bb482e47f35645542c2b768a5fdca5b"}
Sep 10 21:22:38.949601 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:38.949554    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg" podStartSLOduration=1.949540102 podStartE2EDuration="1.949540102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:22:38.947625311 +0000 UTC m=+1499.923285359" watchObservedRunningTime="2026-09-10 21:22:38.949540102 +0000 UTC m=+1499.925200148"
Sep 10 21:22:47.622932 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:47.622890    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:47.623418 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:47.622950    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:47.636150 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:47.636128    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:22:47.981707 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:22:47.981626    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:23:00.937437 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:00.937406    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst_b31d8328-f34d-4d6e-b284-8b6261ec2971/payload-processing/0.log"
Sep 10 21:23:01.069219 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:01.069162    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kvhz2_ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f/payload-processing/0.log"
Sep 10 21:23:07.267791 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:07.267759    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"]
Sep 10 21:23:07.271241 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:07.271222    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"
Sep 10 21:23:07.273588 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:07.273559    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-c1753c-dockercfg-ccsz6\""
Sep 10 21:23:07.273727 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:07.273596    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-c1753c\""
Sep 10 21:23:07.282444 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:07.282419    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"]
Sep 10 21:23:07.356625 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:07.356592    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjcxr\" (UniqueName: \"kubernetes.io/projected/a9eae7a2-c4b3-415d-bd6f-e920d613c428-kube-api-access-fjcxr\") pod \"payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs\" (UID: \"a9eae7a2-c4b3-415d-bd6f-e920d613c428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"
Sep 10 21:23:07.356810 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:07.356719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9eae7a2-c4b3-415d-bd6f-e920d613c428-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs\" (UID: \"a9eae7a2-c4b3-415d-bd6f-e920d613c428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"
Sep 10 21:23:07.457499 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:07.457459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9eae7a2-c4b3-415d-bd6f-e920d613c428-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs\" (UID: \"a9eae7a2-c4b3-415d-bd6f-e920d613c428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"
Sep 10 21:23:07.457676 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:07.457510    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjcxr\" (UniqueName: \"kubernetes.io/projected/a9eae7a2-c4b3-415d-bd6f-e920d613c428-kube-api-access-fjcxr\") pod \"payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs\" (UID: \"a9eae7a2-c4b3-415d-bd6f-e920d613c428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"
Sep 10 21:23:07.458163 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:07.458144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9eae7a2-c4b3-415d-bd6f-e920d613c428-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs\" (UID: \"a9eae7a2-c4b3-415d-bd6f-e920d613c428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"
Sep 10 21:23:07.465901 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:07.465877    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjcxr\" (UniqueName: \"kubernetes.io/projected/a9eae7a2-c4b3-415d-bd6f-e920d613c428-kube-api-access-fjcxr\") pod \"payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs\" (UID: \"a9eae7a2-c4b3-415d-bd6f-e920d613c428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"
Sep 10 21:23:07.582103 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:07.582071    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"
Sep 10 21:23:07.868891 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:07.868821    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"]
Sep 10 21:23:07.878278 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:23:07.878252    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9eae7a2_c4b3_415d_bd6f_e920d613c428.slice/crio-3385778b878e002cdcc253509d3c0596150fc240560488c880092e3912635f4d WatchSource:0}: Error finding container 3385778b878e002cdcc253509d3c0596150fc240560488c880092e3912635f4d: Status 404 returned error can't find the container with id 3385778b878e002cdcc253509d3c0596150fc240560488c880092e3912635f4d
Sep 10 21:23:08.079771 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:08.079737    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs" event={"ID":"a9eae7a2-c4b3-415d-bd6f-e920d613c428","Type":"ContainerStarted","Data":"af0e6f876333c1901c2c1336896e32481b3996976a08346dfe6c7de85bf81be1"}
Sep 10 21:23:08.079936 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:08.079794    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs" event={"ID":"a9eae7a2-c4b3-415d-bd6f-e920d613c428","Type":"ContainerStarted","Data":"3385778b878e002cdcc253509d3c0596150fc240560488c880092e3912635f4d"}
Sep 10 21:23:08.079990 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:08.079942    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"
Sep 10 21:23:08.098303 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:08.098256    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs" podStartSLOduration=1.098241809 podStartE2EDuration="1.098241809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:23:08.097290623 +0000 UTC m=+1529.072950669" watchObservedRunningTime="2026-09-10 21:23:08.098241809 +0000 UTC m=+1529.073901854"
Sep 10 21:23:19.087118 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:19.087086    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"
Sep 10 21:23:37.418062 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:37.418026    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"]
Sep 10 21:23:37.418699 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:37.418297    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs" podUID="a9eae7a2-c4b3-415d-bd6f-e920d613c428" containerName="payload-pre-processing" containerID="cri-o://af0e6f876333c1901c2c1336896e32481b3996976a08346dfe6c7de85bf81be1" gracePeriod=30
Sep 10 21:23:37.663052 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:37.663025    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"
Sep 10 21:23:37.751156 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:37.751075    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjcxr\" (UniqueName: \"kubernetes.io/projected/a9eae7a2-c4b3-415d-bd6f-e920d613c428-kube-api-access-fjcxr\") pod \"a9eae7a2-c4b3-415d-bd6f-e920d613c428\" (UID: \"a9eae7a2-c4b3-415d-bd6f-e920d613c428\") "
Sep 10 21:23:37.751327 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:37.751263    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9eae7a2-c4b3-415d-bd6f-e920d613c428-plugins-config-volume\") pod \"a9eae7a2-c4b3-415d-bd6f-e920d613c428\" (UID: \"a9eae7a2-c4b3-415d-bd6f-e920d613c428\") "
Sep 10 21:23:37.751624 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:37.751595    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9eae7a2-c4b3-415d-bd6f-e920d613c428-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a9eae7a2-c4b3-415d-bd6f-e920d613c428" (UID: "a9eae7a2-c4b3-415d-bd6f-e920d613c428"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:23:37.753278 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:37.753259    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9eae7a2-c4b3-415d-bd6f-e920d613c428-kube-api-access-fjcxr" (OuterVolumeSpecName: "kube-api-access-fjcxr") pod "a9eae7a2-c4b3-415d-bd6f-e920d613c428" (UID: "a9eae7a2-c4b3-415d-bd6f-e920d613c428"). InnerVolumeSpecName "kube-api-access-fjcxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:23:37.852763 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:37.852733    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9eae7a2-c4b3-415d-bd6f-e920d613c428-plugins-config-volume\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:23:37.852763 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:37.852757    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjcxr\" (UniqueName: \"kubernetes.io/projected/a9eae7a2-c4b3-415d-bd6f-e920d613c428-kube-api-access-fjcxr\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:23:38.230992 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:38.230958    2581 generic.go:356] "Generic (PLEG): container finished" podID="a9eae7a2-c4b3-415d-bd6f-e920d613c428" containerID="af0e6f876333c1901c2c1336896e32481b3996976a08346dfe6c7de85bf81be1" exitCode=0
Sep 10 21:23:38.231170 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:38.231014    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs" event={"ID":"a9eae7a2-c4b3-415d-bd6f-e920d613c428","Type":"ContainerDied","Data":"af0e6f876333c1901c2c1336896e32481b3996976a08346dfe6c7de85bf81be1"}
Sep 10 21:23:38.231170 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:38.231053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs" event={"ID":"a9eae7a2-c4b3-415d-bd6f-e920d613c428","Type":"ContainerDied","Data":"3385778b878e002cdcc253509d3c0596150fc240560488c880092e3912635f4d"}
Sep 10 21:23:38.231170 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:38.231072    2581 scope.go:117] "RemoveContainer" containerID="af0e6f876333c1901c2c1336896e32481b3996976a08346dfe6c7de85bf81be1"
Sep 10 21:23:38.231170 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:38.231016    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"
Sep 10 21:23:38.242739 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:38.242718    2581 scope.go:117] "RemoveContainer" containerID="af0e6f876333c1901c2c1336896e32481b3996976a08346dfe6c7de85bf81be1"
Sep 10 21:23:38.243301 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:23:38.243275    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0e6f876333c1901c2c1336896e32481b3996976a08346dfe6c7de85bf81be1\": container with ID starting with af0e6f876333c1901c2c1336896e32481b3996976a08346dfe6c7de85bf81be1 not found: ID does not exist" containerID="af0e6f876333c1901c2c1336896e32481b3996976a08346dfe6c7de85bf81be1"
Sep 10 21:23:38.243410 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:38.243310    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0e6f876333c1901c2c1336896e32481b3996976a08346dfe6c7de85bf81be1"} err="failed to get container status \"af0e6f876333c1901c2c1336896e32481b3996976a08346dfe6c7de85bf81be1\": rpc error: code = NotFound desc = could not find container \"af0e6f876333c1901c2c1336896e32481b3996976a08346dfe6c7de85bf81be1\": container with ID starting with af0e6f876333c1901c2c1336896e32481b3996976a08346dfe6c7de85bf81be1 not found: ID does not exist"
Sep 10 21:23:38.257340 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:38.257320    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"]
Sep 10 21:23:38.259035 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:38.259017    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c1753c-54874dbffd-gbkxs"]
Sep 10 21:23:39.575011 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:23:39.574979    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9eae7a2-c4b3-415d-bd6f-e920d613c428" path="/var/lib/kubelet/pods/a9eae7a2-c4b3-415d-bd6f-e920d613c428/volumes"
Sep 10 21:24:03.889947 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:03.889910    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"]
Sep 10 21:24:03.890407 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:03.890116    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd" podUID="58de574f-05c8-4efc-b4aa-b8de6d07fd41" containerName="maas-api" containerID="cri-o://ea7300f853b29f826f1c74fb0846c8536d52718d2e72d5a74c036cca79e76caa" gracePeriod=30
Sep 10 21:24:03.953704 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:03.953383    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"]
Sep 10 21:24:03.953704 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:03.953657    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst" podUID="b31d8328-f34d-4d6e-b284-8b6261ec2971" containerName="payload-processing" containerID="cri-o://aaaae8b983fe5d252c3e76058a4cff092fb51b107e221cdb835f833cb0e2ac6d" gracePeriod=30
Sep 10 21:24:04.203574 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.203549    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:24:04.229704 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.229681    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"
Sep 10 21:24:04.297021 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.296995    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58de574f-05c8-4efc-b4aa-b8de6d07fd41-metrics-certs\") pod \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\" (UID: \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\") "
Sep 10 21:24:04.297192 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.297047    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckj29\" (UniqueName: \"kubernetes.io/projected/58de574f-05c8-4efc-b4aa-b8de6d07fd41-kube-api-access-ckj29\") pod \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\" (UID: \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\") "
Sep 10 21:24:04.297192 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.297113    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58de574f-05c8-4efc-b4aa-b8de6d07fd41-openshift-service-ca\") pod \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\" (UID: \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\") "
Sep 10 21:24:04.297192 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.297142    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58de574f-05c8-4efc-b4aa-b8de6d07fd41-maas-api-tls\") pod \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\" (UID: \"58de574f-05c8-4efc-b4aa-b8de6d07fd41\") "
Sep 10 21:24:04.297534 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.297497    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58de574f-05c8-4efc-b4aa-b8de6d07fd41-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "58de574f-05c8-4efc-b4aa-b8de6d07fd41" (UID: "58de574f-05c8-4efc-b4aa-b8de6d07fd41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:24:04.299044 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.299023    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58de574f-05c8-4efc-b4aa-b8de6d07fd41-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "58de574f-05c8-4efc-b4aa-b8de6d07fd41" (UID: "58de574f-05c8-4efc-b4aa-b8de6d07fd41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:24:04.299117 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.299051    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58de574f-05c8-4efc-b4aa-b8de6d07fd41-kube-api-access-ckj29" (OuterVolumeSpecName: "kube-api-access-ckj29") pod "58de574f-05c8-4efc-b4aa-b8de6d07fd41" (UID: "58de574f-05c8-4efc-b4aa-b8de6d07fd41"). InnerVolumeSpecName "kube-api-access-ckj29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:24:04.299165 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.299114    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58de574f-05c8-4efc-b4aa-b8de6d07fd41-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "58de574f-05c8-4efc-b4aa-b8de6d07fd41" (UID: "58de574f-05c8-4efc-b4aa-b8de6d07fd41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:24:04.365904 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.365876    2581 generic.go:356] "Generic (PLEG): container finished" podID="58de574f-05c8-4efc-b4aa-b8de6d07fd41" containerID="ea7300f853b29f826f1c74fb0846c8536d52718d2e72d5a74c036cca79e76caa" exitCode=0
Sep 10 21:24:04.366034 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.365920    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"
Sep 10 21:24:04.366034 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.365956    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd" event={"ID":"58de574f-05c8-4efc-b4aa-b8de6d07fd41","Type":"ContainerDied","Data":"ea7300f853b29f826f1c74fb0846c8536d52718d2e72d5a74c036cca79e76caa"}
Sep 10 21:24:04.366034 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.366001    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd" event={"ID":"58de574f-05c8-4efc-b4aa-b8de6d07fd41","Type":"ContainerDied","Data":"b6d686bafebd16738371e9b6b5fe9c3532b6fd27a1d07440948f14255b86348d"}
Sep 10 21:24:04.366034 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.366028    2581 scope.go:117] "RemoveContainer" containerID="ea7300f853b29f826f1c74fb0846c8536d52718d2e72d5a74c036cca79e76caa"
Sep 10 21:24:04.367449 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.367426    2581 generic.go:356] "Generic (PLEG): container finished" podID="b31d8328-f34d-4d6e-b284-8b6261ec2971" containerID="aaaae8b983fe5d252c3e76058a4cff092fb51b107e221cdb835f833cb0e2ac6d" exitCode=0
Sep 10 21:24:04.367561 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.367466    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"
Sep 10 21:24:04.367561 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.367473    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst" event={"ID":"b31d8328-f34d-4d6e-b284-8b6261ec2971","Type":"ContainerDied","Data":"aaaae8b983fe5d252c3e76058a4cff092fb51b107e221cdb835f833cb0e2ac6d"}
Sep 10 21:24:04.367561 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.367511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst" event={"ID":"b31d8328-f34d-4d6e-b284-8b6261ec2971","Type":"ContainerDied","Data":"b6eb1dd23c41579a60c6182d8b25e3528df730a60bf829ad9024708b4369cab5"}
Sep 10 21:24:04.375451 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.375434    2581 scope.go:117] "RemoveContainer" containerID="ea7300f853b29f826f1c74fb0846c8536d52718d2e72d5a74c036cca79e76caa"
Sep 10 21:24:04.375709 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:24:04.375694    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7300f853b29f826f1c74fb0846c8536d52718d2e72d5a74c036cca79e76caa\": container with ID starting with ea7300f853b29f826f1c74fb0846c8536d52718d2e72d5a74c036cca79e76caa not found: ID does not exist" containerID="ea7300f853b29f826f1c74fb0846c8536d52718d2e72d5a74c036cca79e76caa"
Sep 10 21:24:04.375751 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.375717    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7300f853b29f826f1c74fb0846c8536d52718d2e72d5a74c036cca79e76caa"} err="failed to get container status \"ea7300f853b29f826f1c74fb0846c8536d52718d2e72d5a74c036cca79e76caa\": rpc error: code = NotFound desc = could not find container \"ea7300f853b29f826f1c74fb0846c8536d52718d2e72d5a74c036cca79e76caa\": container with ID starting with ea7300f853b29f826f1c74fb0846c8536d52718d2e72d5a74c036cca79e76caa not found: ID does not exist"
Sep 10 21:24:04.375751 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.375732    2581 scope.go:117] "RemoveContainer" containerID="aaaae8b983fe5d252c3e76058a4cff092fb51b107e221cdb835f833cb0e2ac6d"
Sep 10 21:24:04.383182 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.383163    2581 scope.go:117] "RemoveContainer" containerID="aaaae8b983fe5d252c3e76058a4cff092fb51b107e221cdb835f833cb0e2ac6d"
Sep 10 21:24:04.383449 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:24:04.383430    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaaae8b983fe5d252c3e76058a4cff092fb51b107e221cdb835f833cb0e2ac6d\": container with ID starting with aaaae8b983fe5d252c3e76058a4cff092fb51b107e221cdb835f833cb0e2ac6d not found: ID does not exist" containerID="aaaae8b983fe5d252c3e76058a4cff092fb51b107e221cdb835f833cb0e2ac6d"
Sep 10 21:24:04.383499 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.383457    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaaae8b983fe5d252c3e76058a4cff092fb51b107e221cdb835f833cb0e2ac6d"} err="failed to get container status \"aaaae8b983fe5d252c3e76058a4cff092fb51b107e221cdb835f833cb0e2ac6d\": rpc error: code = NotFound desc = could not find container \"aaaae8b983fe5d252c3e76058a4cff092fb51b107e221cdb835f833cb0e2ac6d\": container with ID starting with aaaae8b983fe5d252c3e76058a4cff092fb51b107e221cdb835f833cb0e2ac6d not found: ID does not exist"
Sep 10 21:24:04.387407 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.387387    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"]
Sep 10 21:24:04.389139 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.389120    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7eb29-7f4b946c8d-vm7pd"]
Sep 10 21:24:04.398264 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.398246    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31d8328-f34d-4d6e-b284-8b6261ec2971-plugins-config-volume\") pod \"b31d8328-f34d-4d6e-b284-8b6261ec2971\" (UID: \"b31d8328-f34d-4d6e-b284-8b6261ec2971\") "
Sep 10 21:24:04.398353 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.398325    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgm9m\" (UniqueName: \"kubernetes.io/projected/b31d8328-f34d-4d6e-b284-8b6261ec2971-kube-api-access-sgm9m\") pod \"b31d8328-f34d-4d6e-b284-8b6261ec2971\" (UID: \"b31d8328-f34d-4d6e-b284-8b6261ec2971\") "
Sep 10 21:24:04.398576 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.398554    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31d8328-f34d-4d6e-b284-8b6261ec2971-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b31d8328-f34d-4d6e-b284-8b6261ec2971" (UID: "b31d8328-f34d-4d6e-b284-8b6261ec2971"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:24:04.398633 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.398620    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58de574f-05c8-4efc-b4aa-b8de6d07fd41-maas-api-tls\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:24:04.398690 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.398639    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58de574f-05c8-4efc-b4aa-b8de6d07fd41-metrics-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:24:04.398690 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.398655    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckj29\" (UniqueName: \"kubernetes.io/projected/58de574f-05c8-4efc-b4aa-b8de6d07fd41-kube-api-access-ckj29\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:24:04.398690 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.398668    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58de574f-05c8-4efc-b4aa-b8de6d07fd41-openshift-service-ca\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:24:04.400075 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.400057    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31d8328-f34d-4d6e-b284-8b6261ec2971-kube-api-access-sgm9m" (OuterVolumeSpecName: "kube-api-access-sgm9m") pod "b31d8328-f34d-4d6e-b284-8b6261ec2971" (UID: "b31d8328-f34d-4d6e-b284-8b6261ec2971"). InnerVolumeSpecName "kube-api-access-sgm9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:24:04.499400 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.499336    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31d8328-f34d-4d6e-b284-8b6261ec2971-plugins-config-volume\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:24:04.499400 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.499358    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgm9m\" (UniqueName: \"kubernetes.io/projected/b31d8328-f34d-4d6e-b284-8b6261ec2971-kube-api-access-sgm9m\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:24:04.688787 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.688755    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"]
Sep 10 21:24:04.692828 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:04.692803    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7eb29-85d5bb667-fdvst"]
Sep 10 21:24:05.573803 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:05.573770    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58de574f-05c8-4efc-b4aa-b8de6d07fd41" path="/var/lib/kubelet/pods/58de574f-05c8-4efc-b4aa-b8de6d07fd41/volumes"
Sep 10 21:24:05.574149 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:05.574111    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b31d8328-f34d-4d6e-b284-8b6261ec2971" path="/var/lib/kubelet/pods/b31d8328-f34d-4d6e-b284-8b6261ec2971/volumes"
Sep 10 21:24:09.513769 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:09.513735    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"]
Sep 10 21:24:24.532907 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.532854    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg" podUID="295308a1-23cc-4eac-8832-9c123ed28034" containerName="main" containerID="cri-o://82aea308ee181d07ba5b48a2de8eee4a8bb482e47f35645542c2b768a5fdca5b" gracePeriod=45
Sep 10 21:24:24.783675 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.783649    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:24:24.883075 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.883046    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-tmp-dir\") pod \"295308a1-23cc-4eac-8832-9c123ed28034\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") "
Sep 10 21:24:24.883279 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.883086    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-model-cache\") pod \"295308a1-23cc-4eac-8832-9c123ed28034\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") "
Sep 10 21:24:24.883279 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.883123    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-home\") pod \"295308a1-23cc-4eac-8832-9c123ed28034\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") "
Sep 10 21:24:24.883279 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.883164    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295308a1-23cc-4eac-8832-9c123ed28034-tls-certs\") pod \"295308a1-23cc-4eac-8832-9c123ed28034\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") "
Sep 10 21:24:24.883279 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.883184    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw8md\" (UniqueName: \"kubernetes.io/projected/295308a1-23cc-4eac-8832-9c123ed28034-kube-api-access-mw8md\") pod \"295308a1-23cc-4eac-8832-9c123ed28034\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") "
Sep 10 21:24:24.883511 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.883341    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "295308a1-23cc-4eac-8832-9c123ed28034" (UID: "295308a1-23cc-4eac-8832-9c123ed28034"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:24:24.883511 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.883418    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-model-cache" (OuterVolumeSpecName: "model-cache") pod "295308a1-23cc-4eac-8832-9c123ed28034" (UID: "295308a1-23cc-4eac-8832-9c123ed28034"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:24:24.883511 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.883424    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-dshm\") pod \"295308a1-23cc-4eac-8832-9c123ed28034\" (UID: \"295308a1-23cc-4eac-8832-9c123ed28034\") "
Sep 10 21:24:24.883511 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.883448    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-home" (OuterVolumeSpecName: "home") pod "295308a1-23cc-4eac-8832-9c123ed28034" (UID: "295308a1-23cc-4eac-8832-9c123ed28034"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:24:24.883790 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.883771    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-home\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:24:24.883855 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.883795    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-tmp-dir\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:24:24.883855 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.883808    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-model-cache\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:24:24.885324 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.885294    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-dshm" (OuterVolumeSpecName: "dshm") pod "295308a1-23cc-4eac-8832-9c123ed28034" (UID: "295308a1-23cc-4eac-8832-9c123ed28034"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:24:24.885428 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.885339    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/295308a1-23cc-4eac-8832-9c123ed28034-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "295308a1-23cc-4eac-8832-9c123ed28034" (UID: "295308a1-23cc-4eac-8832-9c123ed28034"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:24:24.885428 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.885351    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295308a1-23cc-4eac-8832-9c123ed28034-kube-api-access-mw8md" (OuterVolumeSpecName: "kube-api-access-mw8md") pod "295308a1-23cc-4eac-8832-9c123ed28034" (UID: "295308a1-23cc-4eac-8832-9c123ed28034"). InnerVolumeSpecName "kube-api-access-mw8md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:24:24.984967 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.984923    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295308a1-23cc-4eac-8832-9c123ed28034-tls-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:24:24.984967 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.984963    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw8md\" (UniqueName: \"kubernetes.io/projected/295308a1-23cc-4eac-8832-9c123ed28034-kube-api-access-mw8md\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:24:24.984967 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:24.984976    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295308a1-23cc-4eac-8832-9c123ed28034-dshm\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:24:25.473390 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:25.473362    2581 generic.go:356] "Generic (PLEG): container finished" podID="295308a1-23cc-4eac-8832-9c123ed28034" containerID="82aea308ee181d07ba5b48a2de8eee4a8bb482e47f35645542c2b768a5fdca5b" exitCode=0
Sep 10 21:24:25.473583 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:25.473401    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"
Sep 10 21:24:25.473583 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:25.473430    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg" event={"ID":"295308a1-23cc-4eac-8832-9c123ed28034","Type":"ContainerDied","Data":"82aea308ee181d07ba5b48a2de8eee4a8bb482e47f35645542c2b768a5fdca5b"}
Sep 10 21:24:25.473583 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:25.473460    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg" event={"ID":"295308a1-23cc-4eac-8832-9c123ed28034","Type":"ContainerDied","Data":"c92c94b5c75ec46503df689ee14c116aa6809986ed2cfb23517c88b794cc6d43"}
Sep 10 21:24:25.473583 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:25.473478    2581 scope.go:117] "RemoveContainer" containerID="82aea308ee181d07ba5b48a2de8eee4a8bb482e47f35645542c2b768a5fdca5b"
Sep 10 21:24:25.483272 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:25.483256    2581 scope.go:117] "RemoveContainer" containerID="82aea308ee181d07ba5b48a2de8eee4a8bb482e47f35645542c2b768a5fdca5b"
Sep 10 21:24:25.483521 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:24:25.483500    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82aea308ee181d07ba5b48a2de8eee4a8bb482e47f35645542c2b768a5fdca5b\": container with ID starting with 82aea308ee181d07ba5b48a2de8eee4a8bb482e47f35645542c2b768a5fdca5b not found: ID does not exist" containerID="82aea308ee181d07ba5b48a2de8eee4a8bb482e47f35645542c2b768a5fdca5b"
Sep 10 21:24:25.483585 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:25.483534    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82aea308ee181d07ba5b48a2de8eee4a8bb482e47f35645542c2b768a5fdca5b"} err="failed to get container status \"82aea308ee181d07ba5b48a2de8eee4a8bb482e47f35645542c2b768a5fdca5b\": rpc error: code = NotFound desc = could not find container \"82aea308ee181d07ba5b48a2de8eee4a8bb482e47f35645542c2b768a5fdca5b\": container with ID starting with 82aea308ee181d07ba5b48a2de8eee4a8bb482e47f35645542c2b768a5fdca5b not found: ID does not exist"
Sep 10 21:24:25.496508 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:25.496485    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"]
Sep 10 21:24:25.507366 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:25.507346    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-f7eb29/ipp-route-f7eb29-kserve-674f894567-85vfg"]
Sep 10 21:24:25.573510 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:24:25.573485    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="295308a1-23cc-4eac-8832-9c123ed28034" path="/var/lib/kubelet/pods/295308a1-23cc-4eac-8832-9c123ed28034/volumes"
Sep 10 21:25:39.763482 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:39.763451    2581 scope.go:117] "RemoveContainer" containerID="6d92f6c026a1c618e62b89f38839d05e2fa8237ee9a3f05b512c5c3eb12e8719"
Sep 10 21:25:54.042610 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.042572    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"]
Sep 10 21:25:54.043233 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.042829    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89" podUID="3a840edb-409e-4e7e-95ff-1bd29a81f614" containerName="payload-processing" containerID="cri-o://eee843771ff737ab852e6f47277f28878f8b45af86c8be4eeee98f3a80efbee7" gracePeriod=30
Sep 10 21:25:54.296685 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.296626    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"
Sep 10 21:25:54.397400 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.397370    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxlvq\" (UniqueName: \"kubernetes.io/projected/3a840edb-409e-4e7e-95ff-1bd29a81f614-kube-api-access-xxlvq\") pod \"3a840edb-409e-4e7e-95ff-1bd29a81f614\" (UID: \"3a840edb-409e-4e7e-95ff-1bd29a81f614\") "
Sep 10 21:25:54.397400 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.397402    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a840edb-409e-4e7e-95ff-1bd29a81f614-plugins-config-volume\") pod \"3a840edb-409e-4e7e-95ff-1bd29a81f614\" (UID: \"3a840edb-409e-4e7e-95ff-1bd29a81f614\") "
Sep 10 21:25:54.397802 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.397780    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a840edb-409e-4e7e-95ff-1bd29a81f614-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a840edb-409e-4e7e-95ff-1bd29a81f614" (UID: "3a840edb-409e-4e7e-95ff-1bd29a81f614"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:25:54.399456 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.399437    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a840edb-409e-4e7e-95ff-1bd29a81f614-kube-api-access-xxlvq" (OuterVolumeSpecName: "kube-api-access-xxlvq") pod "3a840edb-409e-4e7e-95ff-1bd29a81f614" (UID: "3a840edb-409e-4e7e-95ff-1bd29a81f614"). InnerVolumeSpecName "kube-api-access-xxlvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:25:54.498519 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.498490    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxlvq\" (UniqueName: \"kubernetes.io/projected/3a840edb-409e-4e7e-95ff-1bd29a81f614-kube-api-access-xxlvq\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:25:54.498519 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.498516    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a840edb-409e-4e7e-95ff-1bd29a81f614-plugins-config-volume\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:25:54.903739 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.903701    2581 generic.go:356] "Generic (PLEG): container finished" podID="3a840edb-409e-4e7e-95ff-1bd29a81f614" containerID="eee843771ff737ab852e6f47277f28878f8b45af86c8be4eeee98f3a80efbee7" exitCode=0
Sep 10 21:25:54.903911 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.903749    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"
Sep 10 21:25:54.903911 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.903753    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89" event={"ID":"3a840edb-409e-4e7e-95ff-1bd29a81f614","Type":"ContainerDied","Data":"eee843771ff737ab852e6f47277f28878f8b45af86c8be4eeee98f3a80efbee7"}
Sep 10 21:25:54.903911 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.903810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89" event={"ID":"3a840edb-409e-4e7e-95ff-1bd29a81f614","Type":"ContainerDied","Data":"3fe85cd62c6a1d7ae51cc3c1e7b3238a382ad2bb526f8578c9b2f8168c5c877c"}
Sep 10 21:25:54.903911 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.903832    2581 scope.go:117] "RemoveContainer" containerID="eee843771ff737ab852e6f47277f28878f8b45af86c8be4eeee98f3a80efbee7"
Sep 10 21:25:54.917411 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.912746    2581 scope.go:117] "RemoveContainer" containerID="eee843771ff737ab852e6f47277f28878f8b45af86c8be4eeee98f3a80efbee7"
Sep 10 21:25:54.917411 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:25:54.913060    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee843771ff737ab852e6f47277f28878f8b45af86c8be4eeee98f3a80efbee7\": container with ID starting with eee843771ff737ab852e6f47277f28878f8b45af86c8be4eeee98f3a80efbee7 not found: ID does not exist" containerID="eee843771ff737ab852e6f47277f28878f8b45af86c8be4eeee98f3a80efbee7"
Sep 10 21:25:54.917411 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.913094    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee843771ff737ab852e6f47277f28878f8b45af86c8be4eeee98f3a80efbee7"} err="failed to get container status \"eee843771ff737ab852e6f47277f28878f8b45af86c8be4eeee98f3a80efbee7\": rpc error: code = NotFound desc = could not find container \"eee843771ff737ab852e6f47277f28878f8b45af86c8be4eeee98f3a80efbee7\": container with ID starting with eee843771ff737ab852e6f47277f28878f8b45af86c8be4eeee98f3a80efbee7 not found: ID does not exist"
Sep 10 21:25:54.926854 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.926831    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"]
Sep 10 21:25:54.930492 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:54.930472    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-828c7c-674f9dc45-p7s89"]
Sep 10 21:25:55.573908 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:25:55.573866    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a840edb-409e-4e7e-95ff-1bd29a81f614" path="/var/lib/kubelet/pods/3a840edb-409e-4e7e-95ff-1bd29a81f614/volumes"
Sep 10 21:31:37.460464 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.460428    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-767c774f97-v56v8"]
Sep 10 21:31:37.460871 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.460800    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58de574f-05c8-4efc-b4aa-b8de6d07fd41" containerName="maas-api"
Sep 10 21:31:37.460871 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.460810    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58de574f-05c8-4efc-b4aa-b8de6d07fd41" containerName="maas-api"
Sep 10 21:31:37.460871 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.460819    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b31d8328-f34d-4d6e-b284-8b6261ec2971" containerName="payload-processing"
Sep 10 21:31:37.460871 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.460825    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b31d8328-f34d-4d6e-b284-8b6261ec2971" containerName="payload-processing"
Sep 10 21:31:37.460871 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.460837    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a840edb-409e-4e7e-95ff-1bd29a81f614" containerName="payload-processing"
Sep 10 21:31:37.460871 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.460843    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a840edb-409e-4e7e-95ff-1bd29a81f614" containerName="payload-processing"
Sep 10 21:31:37.460871 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.460860    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295308a1-23cc-4eac-8832-9c123ed28034" containerName="main"
Sep 10 21:31:37.460871 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.460865    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="295308a1-23cc-4eac-8832-9c123ed28034" containerName="main"
Sep 10 21:31:37.460871 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.460871    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9eae7a2-c4b3-415d-bd6f-e920d613c428" containerName="payload-pre-processing"
Sep 10 21:31:37.460871 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.460875    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9eae7a2-c4b3-415d-bd6f-e920d613c428" containerName="payload-pre-processing"
Sep 10 21:31:37.461180 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.460946    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a840edb-409e-4e7e-95ff-1bd29a81f614" containerName="payload-processing"
Sep 10 21:31:37.461180 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.460957    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="295308a1-23cc-4eac-8832-9c123ed28034" containerName="main"
Sep 10 21:31:37.461180 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.460963    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9eae7a2-c4b3-415d-bd6f-e920d613c428" containerName="payload-pre-processing"
Sep 10 21:31:37.461180 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.460970    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b31d8328-f34d-4d6e-b284-8b6261ec2971" containerName="payload-processing"
Sep 10 21:31:37.461180 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.460977    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="58de574f-05c8-4efc-b4aa-b8de6d07fd41" containerName="maas-api"
Sep 10 21:31:37.464031 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.464012    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8"
Sep 10 21:31:37.476094 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.476069    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-767c774f97-v56v8"]
Sep 10 21:31:37.575924 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.575896    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mbld\" (UniqueName: \"kubernetes.io/projected/dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30-kube-api-access-5mbld\") pod \"maas-api-767c774f97-v56v8\" (UID: \"dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30\") " pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8"
Sep 10 21:31:37.575924 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.575928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30-maas-api-tls\") pod \"maas-api-767c774f97-v56v8\" (UID: \"dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30\") " pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8"
Sep 10 21:31:37.576118 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.575964    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30-openshift-service-ca\") pod \"maas-api-767c774f97-v56v8\" (UID: \"dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30\") " pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8"
Sep 10 21:31:37.576118 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.576091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30-metrics-certs\") pod \"maas-api-767c774f97-v56v8\" (UID: \"dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30\") " pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8"
Sep 10 21:31:37.677143 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.677115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mbld\" (UniqueName: \"kubernetes.io/projected/dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30-kube-api-access-5mbld\") pod \"maas-api-767c774f97-v56v8\" (UID: \"dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30\") " pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8"
Sep 10 21:31:37.677143 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.677146    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30-maas-api-tls\") pod \"maas-api-767c774f97-v56v8\" (UID: \"dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30\") " pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8"
Sep 10 21:31:37.677348 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.677181    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30-openshift-service-ca\") pod \"maas-api-767c774f97-v56v8\" (UID: \"dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30\") " pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8"
Sep 10 21:31:37.677348 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.677275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30-metrics-certs\") pod \"maas-api-767c774f97-v56v8\" (UID: \"dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30\") " pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8"
Sep 10 21:31:37.677879 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.677849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30-openshift-service-ca\") pod \"maas-api-767c774f97-v56v8\" (UID: \"dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30\") " pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8"
Sep 10 21:31:37.679594 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.679567    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30-metrics-certs\") pod \"maas-api-767c774f97-v56v8\" (UID: \"dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30\") " pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8"
Sep 10 21:31:37.679691 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.679608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30-maas-api-tls\") pod \"maas-api-767c774f97-v56v8\" (UID: \"dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30\") " pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8"
Sep 10 21:31:37.685852 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.685825    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mbld\" (UniqueName: \"kubernetes.io/projected/dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30-kube-api-access-5mbld\") pod \"maas-api-767c774f97-v56v8\" (UID: \"dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30\") " pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8"
Sep 10 21:31:37.774690 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:37.774619    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8"
Sep 10 21:31:38.050783 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:38.050761    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-767c774f97-v56v8"]
Sep 10 21:31:38.053412 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:31:38.053386    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8d5ad3_37ab_48ab_bfc7_bfab27e37c30.slice/crio-c69ea7a22150d6661781af82da9e07227279f94b827a820abd2f513d0c0dec03 WatchSource:0}: Error finding container c69ea7a22150d6661781af82da9e07227279f94b827a820abd2f513d0c0dec03: Status 404 returned error can't find the container with id c69ea7a22150d6661781af82da9e07227279f94b827a820abd2f513d0c0dec03
Sep 10 21:31:38.054824 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:38.054805    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:31:38.502895 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:38.502867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8" event={"ID":"dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30","Type":"ContainerStarted","Data":"c69ea7a22150d6661781af82da9e07227279f94b827a820abd2f513d0c0dec03"}
Sep 10 21:31:39.509156 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:39.509114    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8" event={"ID":"dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30","Type":"ContainerStarted","Data":"d33e22893b074e5a5cb34b67130e0077e1d51619cb8152c17b17270a4e02dc16"}
Sep 10 21:31:39.509557 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:39.509308    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8"
Sep 10 21:31:39.530081 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:39.530040    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8" podStartSLOduration=2.091221235 podStartE2EDuration="2.530025908s" totalImagesPullingTime="438.804673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:31:39.528880097 +0000 UTC m=+2040.504540156" watchObservedRunningTime="2026-09-10 21:31:39.530025908 +0000 UTC m=+2040.505685955"
Sep 10 21:31:45.519830 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:45.519801    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-767c774f97-v56v8"
Sep 10 21:31:45.574235 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:45.574182    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"]
Sep 10 21:31:45.574428 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:45.574399    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx" podUID="da254e82-d544-4632-b350-eb3fa670018c" containerName="maas-api" containerID="cri-o://acc9669f96daa23dd08f3ec4a85b52fc8325513b455fdb8a41ed1a5455091cf7" gracePeriod=30
Sep 10 21:31:46.815650 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:46.815628    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:31:46.855082 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:46.855053    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nc52\" (UniqueName: \"kubernetes.io/projected/da254e82-d544-4632-b350-eb3fa670018c-kube-api-access-9nc52\") pod \"da254e82-d544-4632-b350-eb3fa670018c\" (UID: \"da254e82-d544-4632-b350-eb3fa670018c\") "
Sep 10 21:31:46.855261 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:46.855088    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da254e82-d544-4632-b350-eb3fa670018c-maas-api-tls\") pod \"da254e82-d544-4632-b350-eb3fa670018c\" (UID: \"da254e82-d544-4632-b350-eb3fa670018c\") "
Sep 10 21:31:46.855261 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:46.855117    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da254e82-d544-4632-b350-eb3fa670018c-openshift-service-ca\") pod \"da254e82-d544-4632-b350-eb3fa670018c\" (UID: \"da254e82-d544-4632-b350-eb3fa670018c\") "
Sep 10 21:31:46.855261 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:46.855243    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da254e82-d544-4632-b350-eb3fa670018c-metrics-certs\") pod \"da254e82-d544-4632-b350-eb3fa670018c\" (UID: \"da254e82-d544-4632-b350-eb3fa670018c\") "
Sep 10 21:31:46.855457 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:46.855433    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da254e82-d544-4632-b350-eb3fa670018c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "da254e82-d544-4632-b350-eb3fa670018c" (UID: "da254e82-d544-4632-b350-eb3fa670018c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:31:46.855619 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:46.855602    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da254e82-d544-4632-b350-eb3fa670018c-openshift-service-ca\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:31:46.857173 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:46.857151    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da254e82-d544-4632-b350-eb3fa670018c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "da254e82-d544-4632-b350-eb3fa670018c" (UID: "da254e82-d544-4632-b350-eb3fa670018c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:31:46.857583 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:46.857566    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da254e82-d544-4632-b350-eb3fa670018c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "da254e82-d544-4632-b350-eb3fa670018c" (UID: "da254e82-d544-4632-b350-eb3fa670018c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:31:46.857672 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:46.857612    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da254e82-d544-4632-b350-eb3fa670018c-kube-api-access-9nc52" (OuterVolumeSpecName: "kube-api-access-9nc52") pod "da254e82-d544-4632-b350-eb3fa670018c" (UID: "da254e82-d544-4632-b350-eb3fa670018c"). InnerVolumeSpecName "kube-api-access-9nc52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:31:46.956297 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:46.956220    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nc52\" (UniqueName: \"kubernetes.io/projected/da254e82-d544-4632-b350-eb3fa670018c-kube-api-access-9nc52\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:31:46.956297 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:46.956247    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da254e82-d544-4632-b350-eb3fa670018c-maas-api-tls\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:31:46.956297 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:46.956260    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da254e82-d544-4632-b350-eb3fa670018c-metrics-certs\") on node \"ip-10-0-134-6.ec2.internal\" DevicePath \"\""
Sep 10 21:31:47.549260 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:47.549228    2581 generic.go:356] "Generic (PLEG): container finished" podID="da254e82-d544-4632-b350-eb3fa670018c" containerID="acc9669f96daa23dd08f3ec4a85b52fc8325513b455fdb8a41ed1a5455091cf7" exitCode=0
Sep 10 21:31:47.549470 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:47.549273    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx" event={"ID":"da254e82-d544-4632-b350-eb3fa670018c","Type":"ContainerDied","Data":"acc9669f96daa23dd08f3ec4a85b52fc8325513b455fdb8a41ed1a5455091cf7"}
Sep 10 21:31:47.549470 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:47.549326    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx" event={"ID":"da254e82-d544-4632-b350-eb3fa670018c","Type":"ContainerDied","Data":"2eabb1fb358045b361777dbc97c15eede83840e4d31d2b30cdee3e18d7908204"}
Sep 10 21:31:47.549470 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:47.549335    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"
Sep 10 21:31:47.549470 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:47.549357    2581 scope.go:117] "RemoveContainer" containerID="acc9669f96daa23dd08f3ec4a85b52fc8325513b455fdb8a41ed1a5455091cf7"
Sep 10 21:31:47.559169 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:47.559152    2581 scope.go:117] "RemoveContainer" containerID="acc9669f96daa23dd08f3ec4a85b52fc8325513b455fdb8a41ed1a5455091cf7"
Sep 10 21:31:47.559486 ip-10-0-134-6 kubenswrapper[2581]: E0910 21:31:47.559465    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc9669f96daa23dd08f3ec4a85b52fc8325513b455fdb8a41ed1a5455091cf7\": container with ID starting with acc9669f96daa23dd08f3ec4a85b52fc8325513b455fdb8a41ed1a5455091cf7 not found: ID does not exist" containerID="acc9669f96daa23dd08f3ec4a85b52fc8325513b455fdb8a41ed1a5455091cf7"
Sep 10 21:31:47.559555 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:47.559493    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc9669f96daa23dd08f3ec4a85b52fc8325513b455fdb8a41ed1a5455091cf7"} err="failed to get container status \"acc9669f96daa23dd08f3ec4a85b52fc8325513b455fdb8a41ed1a5455091cf7\": rpc error: code = NotFound desc = could not find container \"acc9669f96daa23dd08f3ec4a85b52fc8325513b455fdb8a41ed1a5455091cf7\": container with ID starting with acc9669f96daa23dd08f3ec4a85b52fc8325513b455fdb8a41ed1a5455091cf7 not found: ID does not exist"
Sep 10 21:31:47.574797 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:47.574769    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"]
Sep 10 21:31:47.574910 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:47.574803    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5867c8d69-j8mzx"]
Sep 10 21:31:49.574195 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:31:49.574156    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da254e82-d544-4632-b350-eb3fa670018c" path="/var/lib/kubelet/pods/da254e82-d544-4632-b350-eb3fa670018c/volumes"
Sep 10 21:32:05.927459 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:05.927365    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d4c889df7-4zq9w_1863177c-3d62-433a-b69d-b2122d699c18/manager/0.log"
Sep 10 21:32:05.933822 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:05.933800    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d4c889df7-4zq9w_1863177c-3d62-433a-b69d-b2122d699c18/copy-manifests/0.log"
Sep 10 21:32:06.701222 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:06.701169    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-767c774f97-v56v8_dd8d5ad3-37ab-48ab-bfc7-bfab27e37c30/maas-api/0.log"
Sep 10 21:32:07.050085 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:07.050055    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-g6c8r_5cb95453-59ad-45eb-aa0c-11262807be37/postgres/0.log"
Sep 10 21:32:08.207582 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:08.207550    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-drx8l_aea7f7ed-8fb3-447c-b5dd-5e5da268a467/kuadrant-console-plugin/0.log"
Sep 10 21:32:09.191281 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:09.191247    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f4cb8c76f-6zxjc_d102d1d2-c7d1-4886-a79a-607e03ef3122/kube-auth-proxy/0.log"
Sep 10 21:32:09.692106 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:09.692066    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kvhz2_ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f/payload-processing/0.log"
Sep 10 21:32:10.441756 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:10.441728    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-52b2j_bf057229-8150-4530-98e4-78e2844478ca/main/0.log"
Sep 10 21:32:10.680166 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:10.680131    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsk29v_60e26963-0401-4190-8e14-c995cec100e9/main/0.log"
Sep 10 21:32:10.798263 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:10.798236    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-ndhj8_f15ece67-3fa7-4f0e-9f49-83abb10cdcaa/main/0.log"
Sep 10 21:32:10.918594 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:10.918568    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-bjx9h_deca99e8-58d2-458f-9f16-9f51ce6f802e/main/0.log"
Sep 10 21:32:19.859607 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:19.859571    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-29b5p_8cefd1c8-a812-4cde-8fda-cc35af6e9833/global-pull-secret-syncer/0.log"
Sep 10 21:32:20.070962 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:20.070928    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v6rlb_a82c7020-9c1d-41a8-97a5-a752e87fba5a/konnectivity-agent/0.log"
Sep 10 21:32:20.122899 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:20.122822    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-6.ec2.internal_fb31595c699a42d1829717a278752f0e/haproxy/0.log"
Sep 10 21:32:24.547817 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:24.547791    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-drx8l_aea7f7ed-8fb3-447c-b5dd-5e5da268a467/kuadrant-console-plugin/0.log"
Sep 10 21:32:26.289251 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:26.289221    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2c83819f-940a-42b0-95a8-92a8d8359de7/alertmanager/0.log"
Sep 10 21:32:26.329190 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:26.329162    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2c83819f-940a-42b0-95a8-92a8d8359de7/config-reloader/0.log"
Sep 10 21:32:26.354895 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:26.354870    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2c83819f-940a-42b0-95a8-92a8d8359de7/kube-rbac-proxy-web/0.log"
Sep 10 21:32:26.404691 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:26.404668    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2c83819f-940a-42b0-95a8-92a8d8359de7/kube-rbac-proxy/0.log"
Sep 10 21:32:26.466376 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:26.466341    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2c83819f-940a-42b0-95a8-92a8d8359de7/kube-rbac-proxy-metric/0.log"
Sep 10 21:32:26.528180 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:26.528152    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2c83819f-940a-42b0-95a8-92a8d8359de7/prom-label-proxy/0.log"
Sep 10 21:32:26.552318 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:26.552252    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2c83819f-940a-42b0-95a8-92a8d8359de7/init-config-reloader/0.log"
Sep 10 21:32:26.616608 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:26.616579    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-vqvpj_ce584ac9-fc93-43da-8cb2-2f7082a2b59a/kube-state-metrics/0.log"
Sep 10 21:32:26.635150 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:26.635127    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-vqvpj_ce584ac9-fc93-43da-8cb2-2f7082a2b59a/kube-rbac-proxy-main/0.log"
Sep 10 21:32:26.657767 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:26.657749    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-vqvpj_ce584ac9-fc93-43da-8cb2-2f7082a2b59a/kube-rbac-proxy-self/0.log"
Sep 10 21:32:26.692232 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:26.692196    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bcbdfdfbf-85x6w_188ff6df-fa09-4db5-a54c-82fb0bb440bc/metrics-server/0.log"
Sep 10 21:32:26.948992 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:26.948912    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s5sx4_015b0f46-2fc7-4c70-8200-f95bc441ac76/node-exporter/0.log"
Sep 10 21:32:26.978091 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:26.978057    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s5sx4_015b0f46-2fc7-4c70-8200-f95bc441ac76/kube-rbac-proxy/0.log"
Sep 10 21:32:27.001301 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:27.001283    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s5sx4_015b0f46-2fc7-4c70-8200-f95bc441ac76/init-textfile/0.log"
Sep 10 21:32:27.133352 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:27.133319    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fee12dc3-35c4-406c-a0cd-0df23feea5f3/prometheus/0.log"
Sep 10 21:32:27.156885 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:27.156865    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fee12dc3-35c4-406c-a0cd-0df23feea5f3/config-reloader/0.log"
Sep 10 21:32:27.196899 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:27.196876    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fee12dc3-35c4-406c-a0cd-0df23feea5f3/thanos-sidecar/0.log"
Sep 10 21:32:27.230140 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:27.230083    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fee12dc3-35c4-406c-a0cd-0df23feea5f3/kube-rbac-proxy-web/0.log"
Sep 10 21:32:27.259870 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:27.259852    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fee12dc3-35c4-406c-a0cd-0df23feea5f3/kube-rbac-proxy/0.log"
Sep 10 21:32:27.285520 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:27.285497    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fee12dc3-35c4-406c-a0cd-0df23feea5f3/kube-rbac-proxy-thanos/0.log"
Sep 10 21:32:27.315368 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:27.315342    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fee12dc3-35c4-406c-a0cd-0df23feea5f3/init-config-reloader/0.log"
Sep 10 21:32:27.406755 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:27.406721    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-gfx2n_1bd0b49b-7271-4d27-8489-b4b7aa6d9746/prometheus-operator-admission-webhook/0.log"
Sep 10 21:32:27.545438 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:27.545410    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-684c958f79-l44l5_c4adbb2b-130d-4a30-beb9-d8e8c438acc9/thanos-query/0.log"
Sep 10 21:32:27.568472 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:27.568447    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-684c958f79-l44l5_c4adbb2b-130d-4a30-beb9-d8e8c438acc9/kube-rbac-proxy-web/0.log"
Sep 10 21:32:27.592167 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:27.592140    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-684c958f79-l44l5_c4adbb2b-130d-4a30-beb9-d8e8c438acc9/kube-rbac-proxy/0.log"
Sep 10 21:32:27.615354 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:27.615330    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-684c958f79-l44l5_c4adbb2b-130d-4a30-beb9-d8e8c438acc9/prom-label-proxy/0.log"
Sep 10 21:32:27.639448 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:27.639426    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-684c958f79-l44l5_c4adbb2b-130d-4a30-beb9-d8e8c438acc9/kube-rbac-proxy-rules/0.log"
Sep 10 21:32:27.662502 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:27.662485    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-684c958f79-l44l5_c4adbb2b-130d-4a30-beb9-d8e8c438acc9/kube-rbac-proxy-metrics/0.log"
Sep 10 21:32:30.193408 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.193373    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"]
Sep 10 21:32:30.193979 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.193960    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da254e82-d544-4632-b350-eb3fa670018c" containerName="maas-api"
Sep 10 21:32:30.194061 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.193982    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="da254e82-d544-4632-b350-eb3fa670018c" containerName="maas-api"
Sep 10 21:32:30.194116 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.194090    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="da254e82-d544-4632-b350-eb3fa670018c" containerName="maas-api"
Sep 10 21:32:30.197283 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.197263    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:30.199568 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.199548    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mst9f\"/\"kube-root-ca.crt\""
Sep 10 21:32:30.200182 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.200170    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mst9f\"/\"openshift-service-ca.crt\""
Sep 10 21:32:30.200242 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.200189    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mst9f\"/\"default-dockercfg-jxlqh\""
Sep 10 21:32:30.207175 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.207155    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"]
Sep 10 21:32:30.253693 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.253667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4-proc\") pod \"perf-node-gather-daemonset-n8j4v\" (UID: \"4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:30.253820 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.253713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k8fd\" (UniqueName: \"kubernetes.io/projected/4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4-kube-api-access-8k8fd\") pod \"perf-node-gather-daemonset-n8j4v\" (UID: \"4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:30.253820 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.253809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4-lib-modules\") pod \"perf-node-gather-daemonset-n8j4v\" (UID: \"4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:30.253933 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.253838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4-podres\") pod \"perf-node-gather-daemonset-n8j4v\" (UID: \"4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:30.253933 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.253860    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4-sys\") pod \"perf-node-gather-daemonset-n8j4v\" (UID: \"4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:30.354988 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.354959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4-lib-modules\") pod \"perf-node-gather-daemonset-n8j4v\" (UID: \"4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:30.355133 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.354993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4-podres\") pod \"perf-node-gather-daemonset-n8j4v\" (UID: \"4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:30.355133 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.355011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4-sys\") pod \"perf-node-gather-daemonset-n8j4v\" (UID: \"4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:30.355133 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.355054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4-proc\") pod \"perf-node-gather-daemonset-n8j4v\" (UID: \"4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:30.355133 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.355077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k8fd\" (UniqueName: \"kubernetes.io/projected/4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4-kube-api-access-8k8fd\") pod \"perf-node-gather-daemonset-n8j4v\" (UID: \"4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:30.355319 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.355148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4-podres\") pod \"perf-node-gather-daemonset-n8j4v\" (UID: \"4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:30.355319 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.355173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4-proc\") pod \"perf-node-gather-daemonset-n8j4v\" (UID: \"4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:30.355319 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.355155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4-sys\") pod \"perf-node-gather-daemonset-n8j4v\" (UID: \"4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:30.355319 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.355147    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4-lib-modules\") pod \"perf-node-gather-daemonset-n8j4v\" (UID: \"4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:30.362850 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.362831    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k8fd\" (UniqueName: \"kubernetes.io/projected/4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4-kube-api-access-8k8fd\") pod \"perf-node-gather-daemonset-n8j4v\" (UID: \"4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:30.423588 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.423560    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-vhxrv_182f6779-fcd7-40da-9002-04a208edf5a4/download-server/0.log"
Sep 10 21:32:30.507173 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.507107    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:30.785659 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:30.785635    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"]
Sep 10 21:32:30.787078 ip-10-0-134-6 kubenswrapper[2581]: W0910 21:32:30.787046    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4bbb4f81_b28d_4cd8_bed4_497b35e3c3b4.slice/crio-0b95c629add97b1a357891c4c3be8528f0334506205f030977c37f4ac53962e4 WatchSource:0}: Error finding container 0b95c629add97b1a357891c4c3be8528f0334506205f030977c37f4ac53962e4: Status 404 returned error can't find the container with id 0b95c629add97b1a357891c4c3be8528f0334506205f030977c37f4ac53962e4
Sep 10 21:32:31.753112 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:31.753079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v" event={"ID":"4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4","Type":"ContainerStarted","Data":"53a4ec52e61819b39417a3340c996cf44e169dffec53ad159a6ee1f20b6e99cf"}
Sep 10 21:32:31.753540 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:31.753125    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v" event={"ID":"4bbb4f81-b28d-4cd8-bed4-497b35e3c3b4","Type":"ContainerStarted","Data":"0b95c629add97b1a357891c4c3be8528f0334506205f030977c37f4ac53962e4"}
Sep 10 21:32:31.753540 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:31.753245    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:31.768718 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:31.768651    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v" podStartSLOduration=1.76863321 podStartE2EDuration="1.76863321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:31.768327092 +0000 UTC m=+2092.743987140" watchObservedRunningTime="2026-09-10 21:32:31.76863321 +0000 UTC m=+2092.744293258"
Sep 10 21:32:32.363251 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:32.363179    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7mplf_9c25e03f-53c0-43c0-ad21-7301f930243f/dns/0.log"
Sep 10 21:32:32.385786 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:32.385762    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7mplf_9c25e03f-53c0-43c0-ad21-7301f930243f/kube-rbac-proxy/0.log"
Sep 10 21:32:32.500997 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:32.500973    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ps8fr_ddb8643c-235c-4694-b9dd-cd72d8986e42/dns-node-resolver/0.log"
Sep 10 21:32:33.080150 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:33.080120    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ccgh9_27105194-1643-4b6a-ae7b-2e0d76967bb6/node-ca/0.log"
Sep 10 21:32:34.421745 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:34.421713    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f4cb8c76f-6zxjc_d102d1d2-c7d1-4886-a79a-607e03ef3122/kube-auth-proxy/0.log"
Sep 10 21:32:34.580631 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:34.580596    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kvhz2_ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f/payload-processing/0.log"
Sep 10 21:32:35.525469 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:35.525431    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c5cks_6d565bb5-5969-45f9-b862-58ccf6248c9a/serve-healthcheck-canary/0.log"
Sep 10 21:32:36.153772 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:36.153730    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dph2p_e7fd3843-b239-4f87-ab91-808f4a31f064/kube-rbac-proxy/0.log"
Sep 10 21:32:36.176619 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:36.176590    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dph2p_e7fd3843-b239-4f87-ab91-808f4a31f064/exporter/0.log"
Sep 10 21:32:36.199168 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:36.199143    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dph2p_e7fd3843-b239-4f87-ab91-808f4a31f064/extractor/0.log"
Sep 10 21:32:37.767134 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:37.767109    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-n8j4v"
Sep 10 21:32:38.925231 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:38.925188    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d4c889df7-4zq9w_1863177c-3d62-433a-b69d-b2122d699c18/manager/0.log"
Sep 10 21:32:38.950857 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:38.950835    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d4c889df7-4zq9w_1863177c-3d62-433a-b69d-b2122d699c18/copy-manifests/0.log"
Sep 10 21:32:40.648071 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:40.648046    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5d99b4bcb4-7cnkj_9a3479bb-9de5-4712-8c98-8f433b5a9317/manager/0.log"
Sep 10 21:32:40.709678 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:40.709649    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hsg2x_2dbcfdbd-a0e1-4833-ad84-2d99a6d3f04b/openshift-lws-operator/0.log"
Sep 10 21:32:40.798096 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:40.798060    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f4cb8c76f-6zxjc_d102d1d2-c7d1-4886-a79a-607e03ef3122/kube-auth-proxy/0.log"
Sep 10 21:32:40.910248 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:40.910133    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kvhz2_ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f/payload-processing/0.log"
Sep 10 21:32:42.691762 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:42.691732    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f4cb8c76f-6zxjc_d102d1d2-c7d1-4886-a79a-607e03ef3122/kube-auth-proxy/0.log"
Sep 10 21:32:42.803834 ip-10-0-134-6 kubenswrapper[2581]: I0910 21:32:42.803801    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kvhz2_ecc0ba86-c132-406c-b14a-2cdc8d3a3c0f/payload-processing/0.log"