Sep 11 11:46:02.695019 ip-10-0-143-220 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 11:46:02.695034 ip-10-0-143-220 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 11:46:02.695043 ip-10-0-143-220 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 11:46:02.695384 ip-10-0-143-220 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 11:46:12.866687 ip-10-0-143-220 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 11:46:12.866710 ip-10-0-143-220 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e3f32c6341324fcfa811c71fc5a0759e --
Sep 11 11:47:57.451882 ip-10-0-143-220 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 11:47:57.904089 ip-10-0-143-220 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 11:47:57.904089 ip-10-0-143-220 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 11:47:57.904089 ip-10-0-143-220 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 11:47:57.904089 ip-10-0-143-220 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 11:47:57.904089 ip-10-0-143-220 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 11:47:57.906157 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.906093    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 11:47:57.910905 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910883    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 11:47:57.910905 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910905    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 11:47:57.910905 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910909    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 11:47:57.910905 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910913    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 11:47:57.910905 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910916    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910919    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910923    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910925    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910928    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910930    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910933    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910936    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910938    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910941    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910943    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910946    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910949    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910953    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910964    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910967    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910970    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910973    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910976    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 11:47:57.911048 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910980    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910984    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910987    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910990    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910993    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910996    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.910999    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911002    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911006    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911009    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911012    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911015    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911018    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911020    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911023    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911028    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911031    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911033    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911036    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911038    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 11:47:57.911531 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911041    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911044    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911047    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911050    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911053    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911056    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911059    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911061    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911065    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911067    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911070    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911072    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911075    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911077    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911080    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911082    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911085    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911088    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911090    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 11:47:57.912028 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911092    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911095    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911097    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911100    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911102    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911105    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911107    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911110    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911113    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911116    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911118    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911121    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911124    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911126    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911129    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911131    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911134    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911136    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911139    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911142    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 11:47:57.912486 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911144    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 11:47:57.913164 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911147    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 11:47:57.913164 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911150    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 11:47:57.913164 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911152    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 11:47:57.913164 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911155    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 11:47:57.913164 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911157    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 11:47:57.913164 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911159    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 11:47:57.913164 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911162    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 11:47:57.913164 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911164    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 11:47:57.913164 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911167    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 11:47:57.913164 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911169    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 11:47:57.913164 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911172    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 11:47:57.913164 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911174    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 11:47:57.913164 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911176    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 11:47:57.913164 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911179    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 11:47:57.913164 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911182    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 11:47:57.913164 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.911184    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 11:47:57.913769 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913753    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 11:47:57.913815 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913780    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 11:47:57.913815 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913787    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 11:47:57.913815 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913792    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 11:47:57.913815 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913797    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 11:47:57.913815 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913803    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 11:47:57.913815 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913813    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 11:47:57.913815 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913818    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 11:47:57.913987 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913822    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 11:47:57.913987 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913826    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 11:47:57.913987 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913851    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 11:47:57.913987 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913966    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 11:47:57.913987 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913976    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 11:47:57.913987 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913980    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 11:47:57.913987 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913984    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 11:47:57.913987 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913989    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913993    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.913998    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914002    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914006    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914010    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914014    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914018    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914023    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914026    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914030    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914034    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914039    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914043    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914046    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914051    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914057    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914063    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914068    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914072    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 11:47:57.914193 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914076    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914093    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914099    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914103    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914108    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914112    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914116    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914120    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914124    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914128    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914132    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914136    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914140    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914144    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914147    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914150    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914152    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914155    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914158    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 11:47:57.914706 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914160    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914163    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914166    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914168    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914171    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914174    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914177    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914179    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914182    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914184    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914187    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914189    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914192    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914194    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914197    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914199    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914202    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914208    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914212    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 11:47:57.915177 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914216    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914219    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914221    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914224    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914226    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914229    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914232    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914234    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914239    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914242    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914244    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914266    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914271    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914275    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914278    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914281    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914284    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914287    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914290    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914293    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 11:47:57.915680 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914295    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914298    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914301    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914304    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914306    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914309    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914699    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914705    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914708    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914711    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914714    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914716    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914719    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914722    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914724    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914727    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914729    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914734    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914737    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 11:47:57.916151 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914740    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914744    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914746    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914750    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914753    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914755    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914758    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914760    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914763    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914765    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914768    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914770    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914773    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914775    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914778    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914780    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914782    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914785    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914787    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914790    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 11:47:57.916646 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914792    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914795    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914797    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914800    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914802    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914805    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914808    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914810    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914813    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914815    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914818    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914821    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914823    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914826    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914829    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914832    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914834    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914837    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914840    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 11:47:57.917171 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914843    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914847    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914850    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914853    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914856    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914859    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914862    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914864    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914867    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914870    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914872    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914875    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914877    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914879    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914882    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914884    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914887    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914889    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914892    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914894    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 11:47:57.917649 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914897    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914899    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914901    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914904    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914906    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914909    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914912    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914915    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914919    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914922    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914925    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914928    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914930    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914933    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914935    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914939    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914943    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914946    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914948    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 11:47:57.918146 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914951    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914953    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914956    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914958    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914960    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914963    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914965    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.914968    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915051    2569 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915097    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915105    2569 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915110    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915114    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915117    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915122    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915126    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915129    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915132    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915135    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915139    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915143    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915146    2569 flags.go:64] FLAG: --cgroup-root=""
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915149    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 11:47:57.918663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915152    2569 flags.go:64] FLAG: --client-ca-file=""
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915155    2569 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915159    2569 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915162    2569 flags.go:64] FLAG: --cluster-domain=""
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915165    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915168    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915171    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915175    2569 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915179    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915182    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915185    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915189    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915192    2569 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915195    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915198    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915201    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915204    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915208    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915211    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915214    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915217    2569 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915221    2569 flags.go:64] FLAG: --enable-server="true"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915224    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915228    2569 flags.go:64] FLAG: --event-burst="100"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915231    2569 flags.go:64] FLAG: --event-qps="50"
Sep 11 11:47:57.919223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915234    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915237    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915240    2569 flags.go:64] FLAG: --eviction-hard=""
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915244    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915247    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915250    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915253    2569 flags.go:64] FLAG: --eviction-soft=""
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915256    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915259    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915262    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915265    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915268    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915271    2569 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915274    2569 flags.go:64] FLAG: --feature-gates=""
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915278    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915281    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915284    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915287    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915290    2569 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915293    2569 flags.go:64] FLAG: --help="false"
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915296    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-143-220.ec2.internal"
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915299    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915302    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 11:47:57.919834 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915305    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915308    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915311    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915314    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915317    2569 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915320    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915323    2569 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915326    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915329    2569 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915332    2569 flags.go:64] FLAG: --kube-reserved=""
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915335    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915338    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915341    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915344    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915347    2569 flags.go:64] FLAG: --lock-file=""
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915349    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915352    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915355    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915360    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915364    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915367    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915370    2569 flags.go:64] FLAG: --logging-format="text"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915373    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915376    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 11:47:57.920418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915379    2569 flags.go:64] FLAG: --manifest-url=""
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915382    2569 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915386    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915389    2569 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915392    2569 flags.go:64] FLAG: --max-pods="110"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915395    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915398    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915401    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915404    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915407    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915410    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915413    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915420    2569 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915425    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915428    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915431    2569 flags.go:64] FLAG: --pod-cidr=""
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915434    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915439    2569 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915442    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915445    2569 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915448    2569 flags.go:64] FLAG: --port="10250"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915451    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915454    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b358594e7cebf106"
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915457    2569 flags.go:64] FLAG: --qos-reserved=""
Sep 11 11:47:57.921017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915460    2569 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915463    2569 flags.go:64] FLAG: --register-node="true"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915465    2569 flags.go:64] FLAG: --register-with-taints=""
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915469    2569 flags.go:64] FLAG: --registry-burst="10"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915472    2569 flags.go:64] FLAG: --registry-qps="5"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915475    2569 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915478    2569 flags.go:64] FLAG: --reserved-memory=""
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915486    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915489    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915492    2569 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915495    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915498    2569 flags.go:64] FLAG: --runonce="false"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915501    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915504    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915507    2569 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915510    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915512    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915515    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915518    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915521    2569 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915524    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915528    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915531    2569 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915534    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915538    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 11:47:57.921626 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915541    2569 flags.go:64] FLAG: --system-cgroups=""
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915543    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915549    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915551    2569 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915555    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915558    2569 flags.go:64] FLAG: --tls-min-version=""
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915561    2569 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915564    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915567    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915570    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915573    2569 flags.go:64] FLAG: --v="2"
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915577    2569 flags.go:64] FLAG: --version="false"
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915581    2569 flags.go:64] FLAG: --vmodule=""
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915597    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.915601    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915709    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915713    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915716    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915720    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915723    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915726    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915729    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915731    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 11:47:57.922260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915734    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915737    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915740    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915744    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915747    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915750    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915753    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915755    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915760    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915763    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915766    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915769    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915772    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915775    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915778    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915780    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915783    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915785    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915788    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 11:47:57.922845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915791    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915793    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915796    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915799    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915802    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915805    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915807    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915810    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915812    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915815    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915818    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915820    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915823    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915825    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915828    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915831    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915835    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915837    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915840    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915843    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 11:47:57.923313 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915846    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915849    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915852    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915854    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915857    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915860    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915862    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915865    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915867    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915870    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915873    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915876    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915878    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915881    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915884    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915886    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915889    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915892    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915895    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 11:47:57.923800 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915897    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915901    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915904    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915907    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915910    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915913    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915915    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915918    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915920    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915924    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915927    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915930    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915933    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915936    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915939    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915941    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915944    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915946    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915949    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 11:47:57.924272 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915951    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 11:47:57.924743 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915955    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 11:47:57.924743 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915959    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 11:47:57.924743 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915970    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 11:47:57.924743 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915973    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 11:47:57.924743 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915976    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 11:47:57.924743 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915978    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 11:47:57.924743 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915981    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 11:47:57.924743 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915983    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 11:47:57.924743 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915986    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 11:47:57.924743 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915988    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 11:47:57.924743 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915992    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 11:47:57.924743 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915995    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 11:47:57.924743 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.915997    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 11:47:57.924743 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.916679    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 11:47:57.924743 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.923578    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.923607    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.924495    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.924504    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924558    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924564    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924567    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924571    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924574    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924577    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924580    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924600    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924606    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924611    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924615    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924618    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924620    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924623    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924626    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 11:47:57.925196 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924629    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924632    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924635    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924637    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924640    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924642    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924645    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924648    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924650    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924654    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924657    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924660    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924663    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924665    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924668    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924671    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924673    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924677    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924681    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924683    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 11:47:57.925761 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924686    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924689    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924691    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924694    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924697    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924699    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924702    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924705    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924707    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924710    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924712    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924715    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924718    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924720    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924723    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924725    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924728    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924730    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924733    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924735    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 11:47:57.926260 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924738    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924740    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924743    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924746    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924749    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924752    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924755    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924758    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924760    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924763    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924766    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924769    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924771    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924774    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924777    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924779    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924782    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924785    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924788    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 11:47:57.926779 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924790    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924793    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924795    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924798    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924801    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924803    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924806    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924810    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924813    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924817    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924819    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924822    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924824    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924827    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924829    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924832    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924835    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924838    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924841    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 11:47:57.927243 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924843    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 11:47:57.927712 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924846    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 11:47:57.927712 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924849    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 11:47:57.927712 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924851    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 11:47:57.927712 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924854    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 11:47:57.927712 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924857    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 11:47:57.927712 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.924862    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 11:47:57.927712 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924969    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 11:47:57.927712 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924972    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 11:47:57.927712 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924975    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 11:47:57.927712 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924978    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 11:47:57.927712 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924982    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 11:47:57.927712 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924985    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 11:47:57.927712 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924988    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 11:47:57.927712 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924990    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 11:47:57.927712 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924993    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924996    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.924998    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925001    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925004    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925006    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925010    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925014    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925017    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925020    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925022    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925025    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925028    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925030    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925033    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925036    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925040    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925043    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925045    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925048    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 11:47:57.928083 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925051    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925053    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925056    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925058    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925061    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925064    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925067    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925069    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925072    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925075    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925077    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925080    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925082    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925085    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925087    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925090    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925092    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925095    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925097    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925100    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 11:47:57.928566 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925102    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925105    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925107    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925110    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925112    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925115    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925117    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925120    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925124    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925126    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925129    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925131    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925134    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925136    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925139    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925141    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925144    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925148    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925151    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 11:47:57.929060 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925154    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925157    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925160    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925163    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925166    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925168    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925171    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925173    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925176    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925178    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925181    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925184    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925186    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925188    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925191    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925194    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925196    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925199    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925202    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925204    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 11:47:57.929522 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925207    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 11:47:57.930061 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925210    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 11:47:57.930061 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925213    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 11:47:57.930061 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925216    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 11:47:57.930061 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925218    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 11:47:57.930061 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925221    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 11:47:57.930061 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925223    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 11:47:57.930061 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925226    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 11:47:57.930061 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925228    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 11:47:57.930061 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925231    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 11:47:57.930061 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925233    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 11:47:57.930061 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:57.925236    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 11:47:57.930061 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.925241    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 11:47:57.930061 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.925341    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 11:47:57.930061 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.927632    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 11:47:57.930409 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.928637    2569 server.go:1019] "Starting client certificate rotation"
Sep 11 11:47:57.930409 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.928744    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 11:47:57.930409 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.928786    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 11:47:57.931458 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.931445    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 11:47:57.933822 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.933723    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 11:47:57.944154 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.944136    2569 log.go:25] "Validated CRI v1 runtime API"
Sep 11 11:47:57.947396 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.947383    2569 log.go:25] "Validated CRI v1 image API"
Sep 11 11:47:57.948241 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.948226    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 11:47:57.953201 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.953182    2569 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8f450e44-ff04-404b-ad55-eac79c73d00b:/dev/nvme0n1p4 b98eb70f-0c47-4946-af15-3d058154a06a:/dev/nvme0n1p3]
Sep 11 11:47:57.953271 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.953200    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 11:47:57.959159 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.959032    2569 manager.go:216] Machine: {Timestamp:2026-09-11 11:47:57.957533104 +0000 UTC m=+0.379401496 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096169 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d36f9636252a968970e0e176732e4 SystemUUID:ec2d36f9-6362-52a9-6897-0e0e176732e4 BootID:e3f32c63-4132-4fcf-a811-c71fc5a0759e Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:af:57:52:be:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:af:57:52:be:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:89:91:41:72:f3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 11:47:57.959159 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.959149    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 11:47:57.959302 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.959211    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 11:47:57.962636 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.962607    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 11:47:57.962762 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.962636    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-220.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 11:47:57.963475 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.963465    2569 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 11:47:57.963475 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.963477    2569 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 11:47:57.963573 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.963484    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 11:47:57.964314 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.964303    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 11:47:57.965397 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.965387    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 11:47:57.966517 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.966501    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 11:47:57.969538 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.969526    2569 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 11:47:57.969653 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.969644    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 11:47:57.969686 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.969676    2569 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 11:47:57.969686 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.969684    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 11:47:57.969754 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.969694    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 11:47:57.969754 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.969703    2569 kubelet.go:398] "Adding apiserver pod source"
Sep 11 11:47:57.969754 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.969710    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 11:47:57.969980 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.969969    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 11:47:57.970020 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.969988    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 11:47:57.973691 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.973670    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 11:47:57.976651 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.976636    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 11:47:57.976726 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.976666    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 11:47:57.979370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.979358    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 11:47:57.979415 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.979376    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 11:47:57.979415 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.979382    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 11:47:57.979415 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.979387    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 11:47:57.979415 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.979396    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 11:47:57.979415 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.979401    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 11:47:57.979415 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.979413    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 11:47:57.979573 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.979418    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 11:47:57.979573 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.979426    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 11:47:57.979573 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.979431    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 11:47:57.979573 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.979439    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 11:47:57.979573 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.979444    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 11:47:57.980404 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.980394    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 11:47:57.980404 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.980403    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 11:47:57.982203 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:57.982172    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 11:47:57.982286 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:57.982275    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-220.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 11:47:57.984036 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.984021    2569 server.go:1268] "Started kubelet"
Sep 11 11:47:57.984107 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.984061    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 11:47:57.984160 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.984091    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 11:47:57.984206 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.984171    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 11:47:57.984737 ip-10-0-143-220 systemd[1]: Started Kubernetes Kubelet.
Sep 11 11:47:57.986661 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.986574    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 11:47:57.989930 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.989870    2569 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 11:47:57.991633 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:57.991614    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 11:47:57.993239 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.993224    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-220.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 11:47:57.994453 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:57.993346    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-220.ec2.internal.18d441c7b6328f77  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-220.ec2.internal,UID:ip-10-0-143-220.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-220.ec2.internal,},FirstTimestamp:2026-09-11 11:47:57.983993719 +0000 UTC m=+0.405862090,LastTimestamp:2026-09-11 11:47:57.983993719 +0000 UTC m=+0.405862090,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-220.ec2.internal,}"
Sep 11 11:47:57.995961 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.995944    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 11:47:57.996648 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.996630    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 11:47:57.996777 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:57.996759    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-220.ec2.internal\" not found"
Sep 11 11:47:57.997411 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.997372    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 11:47:57.997411 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.997382    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 11:47:57.997411 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.997397    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 11:47:57.997568 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.997415    2569 factory.go:55] Registering systemd factory
Sep 11 11:47:57.997568 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.997429    2569 factory.go:223] Registration of the systemd container factory successfully
Sep 11 11:47:57.997568 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.997464    2569 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 11:47:57.997568 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.997474    2569 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 11:47:57.997732 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.997642    2569 factory.go:153] Registering CRI-O factory
Sep 11 11:47:57.997732 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.997653    2569 factory.go:223] Registration of the crio container factory successfully
Sep 11 11:47:57.997732 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.997690    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 11:47:57.997732 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.997710    2569 factory.go:103] Registering Raw factory
Sep 11 11:47:57.997732 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.997722    2569 manager.go:1196] Started watching for new ooms in manager
Sep 11 11:47:57.998023 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:57.998012    2569 manager.go:349] Starting recovery of all containers
Sep 11 11:47:58.000117 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.000092    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-220.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 11:47:58.000281 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.000249    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 11:47:58.008568 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.008551    2569 manager.go:354] Recovery completed
Sep 11 11:47:58.009850 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.009831    2569 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 11 11:47:58.013207 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.013195    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 11:47:58.015861 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.015841    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z79k7"
Sep 11 11:47:58.016736 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.016717    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 11:47:58.016825 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.016752    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 11:47:58.016825 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.016766    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasSufficientPID"
Sep 11 11:47:58.017319 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.017304    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 11:47:58.017319 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.017317    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 11:47:58.017450 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.017335    2569 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 11:47:58.017450 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.017319    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 11:47:58.018249 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.018233    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 11:47:58.018319 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.018267    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 11:47:58.018319 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.018282    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasSufficientPID"
Sep 11 11:47:58.019012 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.018952    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-220.ec2.internal.18d441c7b8262dbb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-220.ec2.internal,UID:ip-10-0-143-220.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-220.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-220.ec2.internal,},FirstTimestamp:2026-09-11 11:47:58.016736699 +0000 UTC m=+0.438605075,LastTimestamp:2026-09-11 11:47:58.016736699 +0000 UTC m=+0.438605075,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-220.ec2.internal,}"
Sep 11 11:47:58.019396 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.019384    2569 policy_none.go:49] "None policy: Start"
Sep 11 11:47:58.019431 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.019400    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 11:47:58.019431 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.019410    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 11:47:58.021312 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.021298    2569 policy_none.go:47] "Start"
Sep 11 11:47:58.022849 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.022830    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z79k7"
Sep 11 11:47:58.030133 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.030079    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-220.ec2.internal.18d441c7b8268553  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-220.ec2.internal,UID:ip-10-0-143-220.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-220.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-220.ec2.internal,},FirstTimestamp:2026-09-11 11:47:58.016759123 +0000 UTC m=+0.438627495,LastTimestamp:2026-09-11 11:47:58.016759123 +0000 UTC m=+0.438627495,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-220.ec2.internal,}"
Sep 11 11:47:58.074083 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.059169    2569 manager.go:337] "Starting Device Plugin manager"
Sep 11 11:47:58.074083 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.059205    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 11:47:58.074083 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.059214    2569 server.go:87] "Starting device plugin registration server"
Sep 11 11:47:58.074083 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.059417    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 11:47:58.074083 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.059429    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 11:47:58.074083 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.059511    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 11:47:58.074083 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.059598    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 11:47:58.074083 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.059608    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 11:47:58.074083 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.060100    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 11:47:58.074083 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.060143    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-220.ec2.internal\" not found"
Sep 11 11:47:58.074083 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.061792    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 11:47:58.074083 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.063001    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 11:47:58.074083 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.063038    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 11:47:58.074083 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.063055    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 11:47:58.074083 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.063092    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 11:47:58.074083 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.065675    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 11:47:58.160205 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.160142    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 11:47:58.160935 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.160920    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 11:47:58.161010 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.160948    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 11:47:58.161010 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.160963    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasSufficientPID"
Sep 11 11:47:58.161010 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.160989    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.164064 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.164047    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-220.ec2.internal"]
Sep 11 11:47:58.164128 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.164119    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 11:47:58.165485 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.165467    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 11:47:58.165578 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.165501    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 11:47:58.165578 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.165516    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasSufficientPID"
Sep 11 11:47:58.167700 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.167688    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 11:47:58.167836 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.167822    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.167884 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.167850    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 11:47:58.168328 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.168313    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 11:47:58.168422 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.168339    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 11:47:58.168422 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.168349    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasSufficientPID"
Sep 11 11:47:58.168422 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.168348    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 11:47:58.168422 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.168373    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 11:47:58.168422 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.168382    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasSufficientPID"
Sep 11 11:47:58.168600 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.168520    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.168600 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.168536    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-220.ec2.internal\": node \"ip-10-0-143-220.ec2.internal\" not found"
Sep 11 11:47:58.170649 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.170635    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.170699 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.170660    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 11:47:58.171238 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.171225    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 11:47:58.171317 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.171249    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 11:47:58.171317 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.171258    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeHasSufficientPID"
Sep 11 11:47:58.185538 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.185522    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-220.ec2.internal\" not found"
Sep 11 11:47:58.198825 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.198803    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4772d2a6d83dd9e80eabcadfa4ef7e19-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal\" (UID: \"4772d2a6d83dd9e80eabcadfa4ef7e19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.198887 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.198827    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4772d2a6d83dd9e80eabcadfa4ef7e19-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal\" (UID: \"4772d2a6d83dd9e80eabcadfa4ef7e19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.198887 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.198842    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52c830ce49e31490f686af3123732f5d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-220.ec2.internal\" (UID: \"52c830ce49e31490f686af3123732f5d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.202200 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.202185    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-220.ec2.internal\" not found" node="ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.206441 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.206426    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-220.ec2.internal\" not found" node="ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.285639 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.285620    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-220.ec2.internal\" not found"
Sep 11 11:47:58.299491 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.299469    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52c830ce49e31490f686af3123732f5d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-220.ec2.internal\" (UID: \"52c830ce49e31490f686af3123732f5d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.299572 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.299493    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4772d2a6d83dd9e80eabcadfa4ef7e19-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal\" (UID: \"4772d2a6d83dd9e80eabcadfa4ef7e19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.299572 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.299508    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4772d2a6d83dd9e80eabcadfa4ef7e19-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal\" (UID: \"4772d2a6d83dd9e80eabcadfa4ef7e19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.299572 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.299563    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4772d2a6d83dd9e80eabcadfa4ef7e19-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal\" (UID: \"4772d2a6d83dd9e80eabcadfa4ef7e19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.299572 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.299566    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52c830ce49e31490f686af3123732f5d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-220.ec2.internal\" (UID: \"52c830ce49e31490f686af3123732f5d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.299725 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.299577    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4772d2a6d83dd9e80eabcadfa4ef7e19-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal\" (UID: \"4772d2a6d83dd9e80eabcadfa4ef7e19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.386031 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.386010    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-220.ec2.internal\" not found"
Sep 11 11:47:58.486567 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.486489    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-220.ec2.internal\" not found"
Sep 11 11:47:58.505926 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.505906    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.510021 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.510006    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.587375 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.587353    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-220.ec2.internal\" not found"
Sep 11 11:47:58.687865 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.687843    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-220.ec2.internal\" not found"
Sep 11 11:47:58.788523 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:58.788499    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-220.ec2.internal\" not found"
Sep 11 11:47:58.826356 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.826336    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 11:47:58.897959 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.897939    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.913128 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.913111    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 11:47:58.915379 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.915364    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-220.ec2.internal"
Sep 11 11:47:58.926898 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.926881    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 11:47:58.929231 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.929218    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 11:47:58.929334 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.929318    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 11:47:58.929396 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.929372    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 11:47:58.972757 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.972730    2569 apiserver.go:52] "Watching apiserver"
Sep 11 11:47:58.992098 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.992077    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 11:47:58.992417 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.992397    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal","openshift-multus/multus-additional-cni-plugins-7q5t4","openshift-multus/multus-psw94","openshift-multus/network-metrics-daemon-2xwtg","openshift-network-operator/iptables-alerter-g7699","kube-system/global-pull-secret-syncer-sm98f","kube-system/konnectivity-agent-vkfwj","kube-system/kube-apiserver-proxy-ip-10-0-143-220.ec2.internal","openshift-dns/node-resolver-vk54k","openshift-network-diagnostics/network-check-target-p72gz","openshift-ovn-kubernetes/ovnkube-node-b6xv8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc","openshift-cluster-node-tuning-operator/tuned-f792v","openshift-image-registry/node-ca-fjbdx"]
Sep 11 11:47:58.995061 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.995045    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vkfwj"
Sep 11 11:47:58.996036 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.996020    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 11:47:58.997221 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.997203    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:58.997893 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.997875    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xls4w\""
Sep 11 11:47:58.998566 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.998549    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 11:47:58.998673 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.998612    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 11:47:58.999461 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:58.999445    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.000182 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.000164    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-662nm\""
Sep 11 11:47:59.000432 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.000415    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 11:47:59.000501 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.000481    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 11:47:59.001443 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.001425    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 11:47:59.001643 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.001622    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hx66l\""
Sep 11 11:47:59.001749 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.001620    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 11:47:59.001879 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.001863    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:47:59.001945 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.001929    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:47:59.002228 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.002198    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 11:47:59.002268 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.002239    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 11:47:59.004015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.003996    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-multus-cni-dir\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.004083 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004026    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c75825cf-1845-48bc-9e52-4c80394fc360-cni-binary-copy\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.004083 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004045    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-var-lib-cni-multus\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.004083 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004062    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c75825cf-1845-48bc-9e52-4c80394fc360-multus-daemon-config\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.004259 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004083    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-run-multus-certs\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.004259 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004119    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-etc-kubernetes\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.004259 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004141    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-var-lib-cni-bin\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.004259 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004189    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8bb6ad09-b9bd-41ef-bf5d-2ae5a3c28aff-agent-certs\") pod \"konnectivity-agent-vkfwj\" (UID: \"8bb6ad09-b9bd-41ef-bf5d-2ae5a3c28aff\") " pod="kube-system/konnectivity-agent-vkfwj"
Sep 11 11:47:59.004259 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004198    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g7699"
Sep 11 11:47:59.004259 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004245    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-sys\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.004534 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004277    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-var-lib-kubelet\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.004534 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004294    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr77w\" (UniqueName: \"kubernetes.io/projected/235f8771-f39a-471a-b452-2f62c27e0ccb-kube-api-access-xr77w\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.004534 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004316    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-multus-socket-dir-parent\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.004534 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004340    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-run-netns\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.004534 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004363    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-hostroot\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.004534 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004388    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-kubernetes\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.004534 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004410    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-sysctl-conf\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.004534 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004450    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-host\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.004534 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004479    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/235f8771-f39a-471a-b452-2f62c27e0ccb-tmp\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.004534 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004503    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-system-cni-dir\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.004534 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004527    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-run-k8s-cni-cncf-io\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.004914 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004543    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-sysconfig\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.004914 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004557    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-multus-conf-dir\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.004914 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004571    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-modprobe-d\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.004914 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004630    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-var-lib-kubelet\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.004914 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004653    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8bb6ad09-b9bd-41ef-bf5d-2ae5a3c28aff-konnectivity-ca\") pod \"konnectivity-agent-vkfwj\" (UID: \"8bb6ad09-b9bd-41ef-bf5d-2ae5a3c28aff\") " pod="kube-system/konnectivity-agent-vkfwj"
Sep 11 11:47:59.004914 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004682    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-sysctl-d\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.004914 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004709    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp9zm\" (UniqueName: \"kubernetes.io/projected/c75825cf-1845-48bc-9e52-4c80394fc360-kube-api-access-kp9zm\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.004914 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004752    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-run\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.004914 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004776    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-lib-modules\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.004914 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004806    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-cnibin\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.004914 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004829    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-systemd\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.004914 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.004855    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-tuned\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.006439 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.006424    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:47:59.006491 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.006476    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm98f" podUID="584f8d60-a99b-4ab3-acbe-c5de6343284f"
Sep 11 11:47:59.006949 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.006931    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 11:47:59.007048 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.006955    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 11:47:59.007048 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.007020    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 11:47:59.007263 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.007245    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 11:47:59.007648 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.007633    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xdxn9\""
Sep 11 11:47:59.008720 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.008702    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vk54k"
Sep 11 11:47:59.011062 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.011045    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fjbdx"
Sep 11 11:47:59.013402 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.013386    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.014014 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.013993    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 11:47:59.014312 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.014297    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 11:47:59.014741 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.014723    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 11:47:59.014919 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.014904    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qk8tc\""
Sep 11 11:47:59.015052 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.015033    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ljqjh\""
Sep 11 11:47:59.015354 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.015314    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 11:47:59.015779 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.015575    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 11:47:59.015779 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.015751    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 11:47:59.016228 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.016207    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:47:59.016446 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.016420    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:47:59.016623 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.016566    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 11:47:59.017499 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.016993    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xrmth\""
Sep 11 11:47:59.020322 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.020303    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.023039 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.023019    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 11:47:59.023725 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.023679    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 11:47:59.023826 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.023807    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 11:47:59.023900 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.023814    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 11:47:59.023957 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.023898    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 11:47:59.024041 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.024023    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 11:47:59.024542 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.024528    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g6p6d\""
Sep 11 11:47:59.026133 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.026112    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.028972 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.028927    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 11:42:58 +0000 UTC" deadline="2028-03-12 14:49:32.849247752 +0000 UTC"
Sep 11 11:47:59.028972 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.028969    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13155h1m33.820280896s"
Sep 11 11:47:59.031792 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.031775    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 11:47:59.031957 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.031941    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 11:47:59.032203 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.032190    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bgmgc\""
Sep 11 11:47:59.032398 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.032384    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 11:47:59.045769 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.045747    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hnjhh"
Sep 11 11:47:59.055838 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.055820    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hnjhh"
Sep 11 11:47:59.098315 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.098298    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 11:47:59.105102 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105082    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9zm\" (UniqueName: \"kubernetes.io/projected/c75825cf-1845-48bc-9e52-4c80394fc360-kube-api-access-kp9zm\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.105214 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105108    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-lib-modules\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.105214 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105127    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72619839-fc9b-42d3-8442-d14c24b423f5-env-overrides\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.105214 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105144    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25ce4b39-e461-46a5-be7e-e60fa1dad8f0-iptables-alerter-script\") pod \"iptables-alerter-g7699\" (UID: \"25ce4b39-e461-46a5-be7e-e60fa1dad8f0\") " pod="openshift-network-operator/iptables-alerter-g7699"
Sep 11 11:47:59.105214 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105166    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret\") pod \"global-pull-secret-syncer-sm98f\" (UID: \"584f8d60-a99b-4ab3-acbe-c5de6343284f\") " pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:47:59.105214 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105190    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc1ebadd-d2ed-4de2-8225-21f60ffae756-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.105214 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105214    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72619839-fc9b-42d3-8442-d14c24b423f5-ovn-node-metrics-cert\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.105480 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105250    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j852v\" (UniqueName: \"kubernetes.io/projected/72619839-fc9b-42d3-8442-d14c24b423f5-kube-api-access-j852v\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.105480 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105266    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-lib-modules\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.105480 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105304    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-cnibin\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.105480 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105339    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-systemd\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.105480 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105365    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-tuned\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.105480 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105374    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-cnibin\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.105480 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105391    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-registration-dir\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.105480 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105416    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-device-dir\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.105480 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105444    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-systemd\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.106015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105508    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc1ebadd-d2ed-4de2-8225-21f60ffae756-cnibin\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.106015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105543    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-var-lib-cni-multus\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.106015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105573    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-var-lib-cni-multus\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.106015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105579    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c75825cf-1845-48bc-9e52-4c80394fc360-multus-daemon-config\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.106015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105625    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-etc-kubernetes\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.106015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105650    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8bb6ad09-b9bd-41ef-bf5d-2ae5a3c28aff-agent-certs\") pod \"konnectivity-agent-vkfwj\" (UID: \"8bb6ad09-b9bd-41ef-bf5d-2ae5a3c28aff\") " pod="kube-system/konnectivity-agent-vkfwj"
Sep 11 11:47:59.106015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105679    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.106015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105704    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72619839-fc9b-42d3-8442-d14c24b423f5-ovnkube-config\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.106015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105721    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-etc-kubernetes\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.106015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105730    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4lzw\" (UniqueName: \"kubernetes.io/projected/25ce4b39-e461-46a5-be7e-e60fa1dad8f0-kube-api-access-p4lzw\") pod \"iptables-alerter-g7699\" (UID: \"25ce4b39-e461-46a5-be7e-e60fa1dad8f0\") " pod="openshift-network-operator/iptables-alerter-g7699"
Sep 11 11:47:59.106015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105756    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-sys-fs\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.106015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105782    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-multus-socket-dir-parent\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.106015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105806    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-hostroot\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.106015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105831    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-kubernetes\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.106015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105858    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-system-cni-dir\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.106015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105883    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-run-k8s-cni-cncf-io\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.106015 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105908    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-systemd-units\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105934    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72619839-fc9b-42d3-8442-d14c24b423f5-ovnkube-script-lib\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105937    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-kubernetes\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105951    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-hostroot\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105958    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-socket-dir\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105937    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105949    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-multus-socket-dir-parent\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105988    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc1ebadd-d2ed-4de2-8225-21f60ffae756-cni-binary-copy\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.105992    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-system-cni-dir\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106017    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-multus-conf-dir\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106057    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-run-netns\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106062    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-multus-conf-dir\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106055    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-run-k8s-cni-cncf-io\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106083    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c75825cf-1845-48bc-9e52-4c80394fc360-multus-daemon-config\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106114    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-sysctl-d\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106154    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106216    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc1ebadd-d2ed-4de2-8225-21f60ffae756-system-cni-dir\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106245    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-sysctl-d\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.106829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106246    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-run\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.107733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106283    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-run-ovn\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.107733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106296    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-run\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.107733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106318    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/584f8d60-a99b-4ab3-acbe-c5de6343284f-kubelet-config\") pod \"global-pull-secret-syncer-sm98f\" (UID: \"584f8d60-a99b-4ab3-acbe-c5de6343284f\") " pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:47:59.107733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106345    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/714f5e6f-c22d-48de-b545-6be2e58968b0-hosts-file\") pod \"node-resolver-vk54k\" (UID: \"714f5e6f-c22d-48de-b545-6be2e58968b0\") " pod="openshift-dns/node-resolver-vk54k"
Sep 11 11:47:59.107733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106375    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-kubelet\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.107733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106399    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-run-systemd\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.107733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106434    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-node-log\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.107733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106479    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-cni-bin\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.107733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106505    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lvzv\" (UniqueName: \"kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv\") pod \"network-check-target-p72gz\" (UID: \"83d5cf45-f64b-4e9d-8d77-088b52928c66\") " pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:47:59.107733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106548    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-multus-cni-dir\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.107733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106573    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c75825cf-1845-48bc-9e52-4c80394fc360-cni-binary-copy\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.107733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106614    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-run-multus-certs\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.107733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106639    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-log-socket\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.107733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106642    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-multus-cni-dir\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.107733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106662    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/584f8d60-a99b-4ab3-acbe-c5de6343284f-dbus\") pod \"global-pull-secret-syncer-sm98f\" (UID: \"584f8d60-a99b-4ab3-acbe-c5de6343284f\") " pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:47:59.107733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106682    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-run-multus-certs\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.107733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106691    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd6q5\" (UniqueName: \"kubernetes.io/projected/714f5e6f-c22d-48de-b545-6be2e58968b0-kube-api-access-qd6q5\") pod \"node-resolver-vk54k\" (UID: \"714f5e6f-c22d-48de-b545-6be2e58968b0\") " pod="openshift-dns/node-resolver-vk54k"
Sep 11 11:47:59.108744 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106717    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf2lq\" (UniqueName: \"kubernetes.io/projected/fc1ebadd-d2ed-4de2-8225-21f60ffae756-kube-api-access-pf2lq\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.108744 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106743    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-var-lib-cni-bin\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.108744 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106783    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-sys\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.108744 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106794    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-var-lib-cni-bin\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.108744 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106807    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-var-lib-kubelet\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.108744 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106838    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-sys\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.108744 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106833    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr77w\" (UniqueName: \"kubernetes.io/projected/235f8771-f39a-471a-b452-2f62c27e0ccb-kube-api-access-xr77w\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.108744 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106878    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.108744 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106892    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-var-lib-kubelet\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.108744 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106905    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjl6v\" (UniqueName: \"kubernetes.io/projected/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-kube-api-access-vjl6v\") pod \"network-metrics-daemon-2xwtg\" (UID: \"ca51005b-bb90-4eac-8bc7-c64bb5463c3f\") " pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:47:59.108744 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106950    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96f46a0c-d972-4490-a888-ca0f79174b50-serviceca\") pod \"node-ca-fjbdx\" (UID: \"96f46a0c-d972-4490-a888-ca0f79174b50\") " pod="openshift-image-registry/node-ca-fjbdx"
Sep 11 11:47:59.108744 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106970    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-run-netns\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.108744 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.106990    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-sysctl-conf\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.108744 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107013    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-host\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.108744 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107036    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/235f8771-f39a-471a-b452-2f62c27e0ccb-tmp\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.108744 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107047    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c75825cf-1845-48bc-9e52-4c80394fc360-cni-binary-copy\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.108744 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107061    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-slash\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.109654 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107067    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-host\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.109654 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107089    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69822\" (UniqueName: \"kubernetes.io/projected/c5ed4785-f3e8-424b-936d-588ebe7179a6-kube-api-access-69822\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.109654 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107101    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-run-netns\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.109654 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107116    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d64px\" (UniqueName: \"kubernetes.io/projected/96f46a0c-d972-4490-a888-ca0f79174b50-kube-api-access-d64px\") pod \"node-ca-fjbdx\" (UID: \"96f46a0c-d972-4490-a888-ca0f79174b50\") " pod="openshift-image-registry/node-ca-fjbdx"
Sep 11 11:47:59.109654 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107141    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-sysconfig\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.109654 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107168    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc1ebadd-d2ed-4de2-8225-21f60ffae756-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.109654 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107166    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-sysctl-conf\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.109654 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107196    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-modprobe-d\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.109654 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107221    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-etc-openvswitch\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.109654 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107226    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-sysconfig\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.109654 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107254    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-run-openvswitch\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.109654 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107278    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-cni-netd\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.109654 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107312    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25ce4b39-e461-46a5-be7e-e60fa1dad8f0-host-slash\") pod \"iptables-alerter-g7699\" (UID: \"25ce4b39-e461-46a5-be7e-e60fa1dad8f0\") " pod="openshift-network-operator/iptables-alerter-g7699"
Sep 11 11:47:59.109654 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107362    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.109654 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107397    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs\") pod \"network-metrics-daemon-2xwtg\" (UID: \"ca51005b-bb90-4eac-8bc7-c64bb5463c3f\") " pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:47:59.109654 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107434    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc1ebadd-d2ed-4de2-8225-21f60ffae756-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.109654 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107485    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-var-lib-kubelet\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.110266 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107511    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8bb6ad09-b9bd-41ef-bf5d-2ae5a3c28aff-konnectivity-ca\") pod \"konnectivity-agent-vkfwj\" (UID: \"8bb6ad09-b9bd-41ef-bf5d-2ae5a3c28aff\") " pod="kube-system/konnectivity-agent-vkfwj"
Sep 11 11:47:59.110266 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107535    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-var-lib-openvswitch\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.110266 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107559    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714f5e6f-c22d-48de-b545-6be2e58968b0-tmp-dir\") pod \"node-resolver-vk54k\" (UID: \"714f5e6f-c22d-48de-b545-6be2e58968b0\") " pod="openshift-dns/node-resolver-vk54k"
Sep 11 11:47:59.110266 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107582    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96f46a0c-d972-4490-a888-ca0f79174b50-host\") pod \"node-ca-fjbdx\" (UID: \"96f46a0c-d972-4490-a888-ca0f79174b50\") " pod="openshift-image-registry/node-ca-fjbdx"
Sep 11 11:47:59.110266 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107601    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c75825cf-1845-48bc-9e52-4c80394fc360-host-var-lib-kubelet\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.110266 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.107325    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-modprobe-d\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.110266 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.108192    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8bb6ad09-b9bd-41ef-bf5d-2ae5a3c28aff-konnectivity-ca\") pod \"konnectivity-agent-vkfwj\" (UID: \"8bb6ad09-b9bd-41ef-bf5d-2ae5a3c28aff\") " pod="kube-system/konnectivity-agent-vkfwj"
Sep 11 11:47:59.110266 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.109758    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/235f8771-f39a-471a-b452-2f62c27e0ccb-tmp\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.110266 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.109842    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/235f8771-f39a-471a-b452-2f62c27e0ccb-etc-tuned\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.110266 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.109942    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8bb6ad09-b9bd-41ef-bf5d-2ae5a3c28aff-agent-certs\") pod \"konnectivity-agent-vkfwj\" (UID: \"8bb6ad09-b9bd-41ef-bf5d-2ae5a3c28aff\") " pod="kube-system/konnectivity-agent-vkfwj"
Sep 11 11:47:59.114581 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.114553    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp9zm\" (UniqueName: \"kubernetes.io/projected/c75825cf-1845-48bc-9e52-4c80394fc360-kube-api-access-kp9zm\") pod \"multus-psw94\" (UID: \"c75825cf-1845-48bc-9e52-4c80394fc360\") " pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.115117 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.115099    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr77w\" (UniqueName: \"kubernetes.io/projected/235f8771-f39a-471a-b452-2f62c27e0ccb-kube-api-access-xr77w\") pod \"tuned-f792v\" (UID: \"235f8771-f39a-471a-b452-2f62c27e0ccb\") " pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.207803 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.207772    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc1ebadd-d2ed-4de2-8225-21f60ffae756-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.207969 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.207805    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-etc-openvswitch\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.207969 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.207827    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-run-openvswitch\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.207969 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.207848    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-cni-netd\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.207969 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.207902    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-cni-netd\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.207969 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.207910    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-etc-openvswitch\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.207969 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.207924    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-run-openvswitch\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.207969 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.207940    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25ce4b39-e461-46a5-be7e-e60fa1dad8f0-host-slash\") pod \"iptables-alerter-g7699\" (UID: \"25ce4b39-e461-46a5-be7e-e60fa1dad8f0\") " pod="openshift-network-operator/iptables-alerter-g7699"
Sep 11 11:47:59.207969 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.207965    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.208418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.207981    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs\") pod \"network-metrics-daemon-2xwtg\" (UID: \"ca51005b-bb90-4eac-8bc7-c64bb5463c3f\") " pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:47:59.208418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208000    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc1ebadd-d2ed-4de2-8225-21f60ffae756-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.208418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208023    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-var-lib-openvswitch\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.208418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208070    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-var-lib-openvswitch\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.208418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208102    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.208418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208106    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25ce4b39-e461-46a5-be7e-e60fa1dad8f0-host-slash\") pod \"iptables-alerter-g7699\" (UID: \"25ce4b39-e461-46a5-be7e-e60fa1dad8f0\") " pod="openshift-network-operator/iptables-alerter-g7699"
Sep 11 11:47:59.208418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208155    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714f5e6f-c22d-48de-b545-6be2e58968b0-tmp-dir\") pod \"node-resolver-vk54k\" (UID: \"714f5e6f-c22d-48de-b545-6be2e58968b0\") " pod="openshift-dns/node-resolver-vk54k"
Sep 11 11:47:59.208418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208187    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96f46a0c-d972-4490-a888-ca0f79174b50-host\") pod \"node-ca-fjbdx\" (UID: \"96f46a0c-d972-4490-a888-ca0f79174b50\") " pod="openshift-image-registry/node-ca-fjbdx"
Sep 11 11:47:59.208418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208214    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72619839-fc9b-42d3-8442-d14c24b423f5-env-overrides\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.208418 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.208221    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 11:47:59.208418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208243    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25ce4b39-e461-46a5-be7e-e60fa1dad8f0-iptables-alerter-script\") pod \"iptables-alerter-g7699\" (UID: \"25ce4b39-e461-46a5-be7e-e60fa1dad8f0\") " pod="openshift-network-operator/iptables-alerter-g7699"
Sep 11 11:47:59.208418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208251    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96f46a0c-d972-4490-a888-ca0f79174b50-host\") pod \"node-ca-fjbdx\" (UID: \"96f46a0c-d972-4490-a888-ca0f79174b50\") " pod="openshift-image-registry/node-ca-fjbdx"
Sep 11 11:47:59.208418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208271    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret\") pod \"global-pull-secret-syncer-sm98f\" (UID: \"584f8d60-a99b-4ab3-acbe-c5de6343284f\") " pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:47:59.208418 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.208311    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs podName:ca51005b-bb90-4eac-8bc7-c64bb5463c3f nodeName:}" failed. No retries permitted until 2026-09-11 11:47:59.708286655 +0000 UTC m=+2.130155039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs") pod "network-metrics-daemon-2xwtg" (UID: "ca51005b-bb90-4eac-8bc7-c64bb5463c3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 11:47:59.208418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208338    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc1ebadd-d2ed-4de2-8225-21f60ffae756-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.208418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208372    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72619839-fc9b-42d3-8442-d14c24b423f5-ovn-node-metrics-cert\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.208418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208399    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j852v\" (UniqueName: \"kubernetes.io/projected/72619839-fc9b-42d3-8442-d14c24b423f5-kube-api-access-j852v\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.209370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208427    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-registration-dir\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.209370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208424    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc1ebadd-d2ed-4de2-8225-21f60ffae756-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.209370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208524    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714f5e6f-c22d-48de-b545-6be2e58968b0-tmp-dir\") pod \"node-resolver-vk54k\" (UID: \"714f5e6f-c22d-48de-b545-6be2e58968b0\") " pod="openshift-dns/node-resolver-vk54k"
Sep 11 11:47:59.209370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208602    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc1ebadd-d2ed-4de2-8225-21f60ffae756-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.209370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208666    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-device-dir\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.209370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208699    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc1ebadd-d2ed-4de2-8225-21f60ffae756-cnibin\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.209370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208721    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-registration-dir\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.209370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208728    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.209370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208765    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72619839-fc9b-42d3-8442-d14c24b423f5-ovnkube-config\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.209370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208769    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.209370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208662    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc1ebadd-d2ed-4de2-8225-21f60ffae756-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.209370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208797    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4lzw\" (UniqueName: \"kubernetes.io/projected/25ce4b39-e461-46a5-be7e-e60fa1dad8f0-kube-api-access-p4lzw\") pod \"iptables-alerter-g7699\" (UID: \"25ce4b39-e461-46a5-be7e-e60fa1dad8f0\") " pod="openshift-network-operator/iptables-alerter-g7699"
Sep 11 11:47:59.209370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208802    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72619839-fc9b-42d3-8442-d14c24b423f5-env-overrides\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.209370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208828    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-sys-fs\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.209370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208841    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc1ebadd-d2ed-4de2-8225-21f60ffae756-cnibin\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.209370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208802    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-device-dir\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.209370 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.208887    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 11:47:59.210287 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208908    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-systemd-units\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.210287 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208912    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25ce4b39-e461-46a5-be7e-e60fa1dad8f0-iptables-alerter-script\") pod \"iptables-alerter-g7699\" (UID: \"25ce4b39-e461-46a5-be7e-e60fa1dad8f0\") " pod="openshift-network-operator/iptables-alerter-g7699"
Sep 11 11:47:59.210287 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208886    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-sys-fs\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.210287 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.208938    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret podName:584f8d60-a99b-4ab3-acbe-c5de6343284f nodeName:}" failed. No retries permitted until 2026-09-11 11:47:59.708921118 +0000 UTC m=+2.130789496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret") pod "global-pull-secret-syncer-sm98f" (UID: "584f8d60-a99b-4ab3-acbe-c5de6343284f") : object "kube-system"/"original-pull-secret" not registered
Sep 11 11:47:59.210287 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208954    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-systemd-units\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.210287 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208959    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72619839-fc9b-42d3-8442-d14c24b423f5-ovnkube-script-lib\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.210287 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.208993    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-socket-dir\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.210287 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209021    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc1ebadd-d2ed-4de2-8225-21f60ffae756-cni-binary-copy\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.210287 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209047    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-run-netns\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.210287 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209073    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.210287 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209108    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc1ebadd-d2ed-4de2-8225-21f60ffae756-system-cni-dir\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.210287 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209110    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-socket-dir\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.210287 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209138    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-run-ovn\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.210287 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209167    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/584f8d60-a99b-4ab3-acbe-c5de6343284f-kubelet-config\") pod \"global-pull-secret-syncer-sm98f\" (UID: \"584f8d60-a99b-4ab3-acbe-c5de6343284f\") " pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:47:59.210287 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209183    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5ed4785-f3e8-424b-936d-588ebe7179a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.210287 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209192    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/714f5e6f-c22d-48de-b545-6be2e58968b0-hosts-file\") pod \"node-resolver-vk54k\" (UID: \"714f5e6f-c22d-48de-b545-6be2e58968b0\") " pod="openshift-dns/node-resolver-vk54k"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209211    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc1ebadd-d2ed-4de2-8225-21f60ffae756-system-cni-dir\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209228    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-kubelet\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209247    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/714f5e6f-c22d-48de-b545-6be2e58968b0-hosts-file\") pod \"node-resolver-vk54k\" (UID: \"714f5e6f-c22d-48de-b545-6be2e58968b0\") " pod="openshift-dns/node-resolver-vk54k"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209258    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-run-systemd\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209274    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/584f8d60-a99b-4ab3-acbe-c5de6343284f-kubelet-config\") pod \"global-pull-secret-syncer-sm98f\" (UID: \"584f8d60-a99b-4ab3-acbe-c5de6343284f\") " pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209289    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-node-log\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209300    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-run-ovn\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209165    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-run-netns\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209319    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-run-systemd\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209314    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-cni-bin\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209353    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-cni-bin\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209362    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-kubelet\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209367    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvzv\" (UniqueName: \"kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv\") pod \"network-check-target-p72gz\" (UID: \"83d5cf45-f64b-4e9d-8d77-088b52928c66\") " pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209400    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-log-socket\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209426    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/584f8d60-a99b-4ab3-acbe-c5de6343284f-dbus\") pod \"global-pull-secret-syncer-sm98f\" (UID: \"584f8d60-a99b-4ab3-acbe-c5de6343284f\") " pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209424    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72619839-fc9b-42d3-8442-d14c24b423f5-ovnkube-config\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209453    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd6q5\" (UniqueName: \"kubernetes.io/projected/714f5e6f-c22d-48de-b545-6be2e58968b0-kube-api-access-qd6q5\") pod \"node-resolver-vk54k\" (UID: \"714f5e6f-c22d-48de-b545-6be2e58968b0\") " pod="openshift-dns/node-resolver-vk54k"
Sep 11 11:47:59.211188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209466    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72619839-fc9b-42d3-8442-d14c24b423f5-ovnkube-script-lib\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211853 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209475    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-log-socket\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211853 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209402    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-node-log\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211853 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209511    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc1ebadd-d2ed-4de2-8225-21f60ffae756-cni-binary-copy\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.211853 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209513    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf2lq\" (UniqueName: \"kubernetes.io/projected/fc1ebadd-d2ed-4de2-8225-21f60ffae756-kube-api-access-pf2lq\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.211853 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209553    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/584f8d60-a99b-4ab3-acbe-c5de6343284f-dbus\") pod \"global-pull-secret-syncer-sm98f\" (UID: \"584f8d60-a99b-4ab3-acbe-c5de6343284f\") " pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:47:59.211853 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209559    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211853 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209591    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjl6v\" (UniqueName: \"kubernetes.io/projected/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-kube-api-access-vjl6v\") pod \"network-metrics-daemon-2xwtg\" (UID: \"ca51005b-bb90-4eac-8bc7-c64bb5463c3f\") " pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:47:59.211853 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209616    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96f46a0c-d972-4490-a888-ca0f79174b50-serviceca\") pod \"node-ca-fjbdx\" (UID: \"96f46a0c-d972-4490-a888-ca0f79174b50\") " pod="openshift-image-registry/node-ca-fjbdx"
Sep 11 11:47:59.211853 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209625    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211853 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209660    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-slash\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211853 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209705    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72619839-fc9b-42d3-8442-d14c24b423f5-host-slash\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.211853 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209708    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69822\" (UniqueName: \"kubernetes.io/projected/c5ed4785-f3e8-424b-936d-588ebe7179a6-kube-api-access-69822\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.211853 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.209730    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d64px\" (UniqueName: \"kubernetes.io/projected/96f46a0c-d972-4490-a888-ca0f79174b50-kube-api-access-d64px\") pod \"node-ca-fjbdx\" (UID: \"96f46a0c-d972-4490-a888-ca0f79174b50\") " pod="openshift-image-registry/node-ca-fjbdx"
Sep 11 11:47:59.211853 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.210067    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96f46a0c-d972-4490-a888-ca0f79174b50-serviceca\") pod \"node-ca-fjbdx\" (UID: \"96f46a0c-d972-4490-a888-ca0f79174b50\") " pod="openshift-image-registry/node-ca-fjbdx"
Sep 11 11:47:59.211853 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.211258    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72619839-fc9b-42d3-8442-d14c24b423f5-ovn-node-metrics-cert\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.222016 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.221996    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 11:47:59.222016 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.222018    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 11:47:59.222110 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.222028    2569 projected.go:196] Error preparing data for projected volume kube-api-access-9lvzv for pod openshift-network-diagnostics/network-check-target-p72gz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 11:47:59.222110 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.222077    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv podName:83d5cf45-f64b-4e9d-8d77-088b52928c66 nodeName:}" failed. No retries permitted until 2026-09-11 11:47:59.722066077 +0000 UTC m=+2.143934467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9lvzv" (UniqueName: "kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv") pod "network-check-target-p72gz" (UID: "83d5cf45-f64b-4e9d-8d77-088b52928c66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 11:47:59.226529 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.226504    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j852v\" (UniqueName: \"kubernetes.io/projected/72619839-fc9b-42d3-8442-d14c24b423f5-kube-api-access-j852v\") pod \"ovnkube-node-b6xv8\" (UID: \"72619839-fc9b-42d3-8442-d14c24b423f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.227490 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.227466    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4lzw\" (UniqueName: \"kubernetes.io/projected/25ce4b39-e461-46a5-be7e-e60fa1dad8f0-kube-api-access-p4lzw\") pod \"iptables-alerter-g7699\" (UID: \"25ce4b39-e461-46a5-be7e-e60fa1dad8f0\") " pod="openshift-network-operator/iptables-alerter-g7699"
Sep 11 11:47:59.227972 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.227950    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd6q5\" (UniqueName: \"kubernetes.io/projected/714f5e6f-c22d-48de-b545-6be2e58968b0-kube-api-access-qd6q5\") pod \"node-resolver-vk54k\" (UID: \"714f5e6f-c22d-48de-b545-6be2e58968b0\") " pod="openshift-dns/node-resolver-vk54k"
Sep 11 11:47:59.228551 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.228517    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 11:47:59.228696 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.228677    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69822\" (UniqueName: \"kubernetes.io/projected/c5ed4785-f3e8-424b-936d-588ebe7179a6-kube-api-access-69822\") pod \"aws-ebs-csi-driver-node-4wvzc\" (UID: \"c5ed4785-f3e8-424b-936d-588ebe7179a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.229049 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.229031    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjl6v\" (UniqueName: \"kubernetes.io/projected/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-kube-api-access-vjl6v\") pod \"network-metrics-daemon-2xwtg\" (UID: \"ca51005b-bb90-4eac-8bc7-c64bb5463c3f\") " pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:47:59.229172 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.229156    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d64px\" (UniqueName: \"kubernetes.io/projected/96f46a0c-d972-4490-a888-ca0f79174b50-kube-api-access-d64px\") pod \"node-ca-fjbdx\" (UID: \"96f46a0c-d972-4490-a888-ca0f79174b50\") " pod="openshift-image-registry/node-ca-fjbdx"
Sep 11 11:47:59.229261 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.229245    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf2lq\" (UniqueName: \"kubernetes.io/projected/fc1ebadd-d2ed-4de2-8225-21f60ffae756-kube-api-access-pf2lq\") pod \"multus-additional-cni-plugins-7q5t4\" (UID: \"fc1ebadd-d2ed-4de2-8225-21f60ffae756\") " pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.257651 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:59.257628    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4772d2a6d83dd9e80eabcadfa4ef7e19.slice/crio-4d10b1c2aabace40e4ebfd6cfae4258eca42462169e5bd53745a5dc0db086715 WatchSource:0}: Error finding container 4d10b1c2aabace40e4ebfd6cfae4258eca42462169e5bd53745a5dc0db086715: Status 404 returned error can't find the container with id 4d10b1c2aabace40e4ebfd6cfae4258eca42462169e5bd53745a5dc0db086715
Sep 11 11:47:59.264200 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.264186    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 11:47:59.269089 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.269071    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 11:47:59.329476 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.329449    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vkfwj"
Sep 11 11:47:59.341845 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:59.341820    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb6ad09_b9bd_41ef_bf5d_2ae5a3c28aff.slice/crio-f2b107284ef2738bf14ce1256ac5ba186d6cd7754db80348eae0b70ae600656e WatchSource:0}: Error finding container f2b107284ef2738bf14ce1256ac5ba186d6cd7754db80348eae0b70ae600656e: Status 404 returned error can't find the container with id f2b107284ef2738bf14ce1256ac5ba186d6cd7754db80348eae0b70ae600656e
Sep 11 11:47:59.344914 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.344898    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f792v"
Sep 11 11:47:59.353150 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:59.353131    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235f8771_f39a_471a_b452_2f62c27e0ccb.slice/crio-264029e9acf40d5ab141f721474e6d43122247c6347d9ee6d86307ba6b915c01 WatchSource:0}: Error finding container 264029e9acf40d5ab141f721474e6d43122247c6347d9ee6d86307ba6b915c01: Status 404 returned error can't find the container with id 264029e9acf40d5ab141f721474e6d43122247c6347d9ee6d86307ba6b915c01
Sep 11 11:47:59.365247 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.365233    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-psw94"
Sep 11 11:47:59.371312 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.371295    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g7699"
Sep 11 11:47:59.373374 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:59.373348    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75825cf_1845_48bc_9e52_4c80394fc360.slice/crio-993589a38ca72a0df0507c199ec93296d92292dac97ab3d20fcceec984ca39c3 WatchSource:0}: Error finding container 993589a38ca72a0df0507c199ec93296d92292dac97ab3d20fcceec984ca39c3: Status 404 returned error can't find the container with id 993589a38ca72a0df0507c199ec93296d92292dac97ab3d20fcceec984ca39c3
Sep 11 11:47:59.380227 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:59.380200    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ce4b39_e461_46a5_be7e_e60fa1dad8f0.slice/crio-71d890d58242780a35a1e0f99b92c4dd677ecb0d68d0217d57c13bebe7a8df91 WatchSource:0}: Error finding container 71d890d58242780a35a1e0f99b92c4dd677ecb0d68d0217d57c13bebe7a8df91: Status 404 returned error can't find the container with id 71d890d58242780a35a1e0f99b92c4dd677ecb0d68d0217d57c13bebe7a8df91
Sep 11 11:47:59.387631 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.387614    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vk54k"
Sep 11 11:47:59.396413 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:59.396391    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714f5e6f_c22d_48de_b545_6be2e58968b0.slice/crio-7b70ab6e37aebf84721db0e69393e71f230caa93ad047ca3a3420445219b987d WatchSource:0}: Error finding container 7b70ab6e37aebf84721db0e69393e71f230caa93ad047ca3a3420445219b987d: Status 404 returned error can't find the container with id 7b70ab6e37aebf84721db0e69393e71f230caa93ad047ca3a3420445219b987d
Sep 11 11:47:59.411103 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.411087    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fjbdx"
Sep 11 11:47:59.419013 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:59.418994    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f46a0c_d972_4490_a888_ca0f79174b50.slice/crio-2fff8442c82f860bbe92653ed6a1813edd31b7352958cbc8803b8222e0ea23ed WatchSource:0}: Error finding container 2fff8442c82f860bbe92653ed6a1813edd31b7352958cbc8803b8222e0ea23ed: Status 404 returned error can't find the container with id 2fff8442c82f860bbe92653ed6a1813edd31b7352958cbc8803b8222e0ea23ed
Sep 11 11:47:59.433596 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.433576    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7q5t4"
Sep 11 11:47:59.441361 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:59.441342    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1ebadd_d2ed_4de2_8225_21f60ffae756.slice/crio-0770d2e0e6a6bc7f29c8b2eb0161fe24431205353a4624749f85cdced7ed28e4 WatchSource:0}: Error finding container 0770d2e0e6a6bc7f29c8b2eb0161fe24431205353a4624749f85cdced7ed28e4: Status 404 returned error can't find the container with id 0770d2e0e6a6bc7f29c8b2eb0161fe24431205353a4624749f85cdced7ed28e4
Sep 11 11:47:59.458298 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.458271    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:47:59.463385 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.463369    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc"
Sep 11 11:47:59.466323 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:59.466301    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72619839_fc9b_42d3_8442_d14c24b423f5.slice/crio-a41439e6755c9a27fce602e326d46332ce5fdb0cd87b8570426b792aebcbbbc2 WatchSource:0}: Error finding container a41439e6755c9a27fce602e326d46332ce5fdb0cd87b8570426b792aebcbbbc2: Status 404 returned error can't find the container with id a41439e6755c9a27fce602e326d46332ce5fdb0cd87b8570426b792aebcbbbc2
Sep 11 11:47:59.471465 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:59.471441    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ed4785_f3e8_424b_936d_588ebe7179a6.slice/crio-3bc0dbdd2c8fe113f34f84fc671e9505cdfeafc51cb64fa701f6791625c52674 WatchSource:0}: Error finding container 3bc0dbdd2c8fe113f34f84fc671e9505cdfeafc51cb64fa701f6791625c52674: Status 404 returned error can't find the container with id 3bc0dbdd2c8fe113f34f84fc671e9505cdfeafc51cb64fa701f6791625c52674
Sep 11 11:47:59.714933 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.714839    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs\") pod \"network-metrics-daemon-2xwtg\" (UID: \"ca51005b-bb90-4eac-8bc7-c64bb5463c3f\") " pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:47:59.714933 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.714899    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret\") pod \"global-pull-secret-syncer-sm98f\" (UID: \"584f8d60-a99b-4ab3-acbe-c5de6343284f\") " pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:47:59.715123 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.715088    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 11:47:59.715179 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.715158    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret podName:584f8d60-a99b-4ab3-acbe-c5de6343284f nodeName:}" failed. No retries permitted until 2026-09-11 11:48:00.715138327 +0000 UTC m=+3.137006708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret") pod "global-pull-secret-syncer-sm98f" (UID: "584f8d60-a99b-4ab3-acbe-c5de6343284f") : object "kube-system"/"original-pull-secret" not registered
Sep 11 11:47:59.717245 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.715649    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 11:47:59.717245 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.715727    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs podName:ca51005b-bb90-4eac-8bc7-c64bb5463c3f nodeName:}" failed. No retries permitted until 2026-09-11 11:48:00.71570339 +0000 UTC m=+3.137571766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs") pod "network-metrics-daemon-2xwtg" (UID: "ca51005b-bb90-4eac-8bc7-c64bb5463c3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 11:47:59.816218 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:47:59.816187    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvzv\" (UniqueName: \"kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv\") pod \"network-check-target-p72gz\" (UID: \"83d5cf45-f64b-4e9d-8d77-088b52928c66\") " pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:47:59.816344 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.816323    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 11:47:59.816344 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.816338    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 11:47:59.816477 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.816349    2569 projected.go:196] Error preparing data for projected volume kube-api-access-9lvzv for pod openshift-network-diagnostics/network-check-target-p72gz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 11:47:59.816477 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:47:59.816395    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv podName:83d5cf45-f64b-4e9d-8d77-088b52928c66 nodeName:}" failed. No retries permitted until 2026-09-11 11:48:00.816378848 +0000 UTC m=+3.238247228 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9lvzv" (UniqueName: "kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv") pod "network-check-target-p72gz" (UID: "83d5cf45-f64b-4e9d-8d77-088b52928c66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 11:47:59.826317 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:47:59.826290    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c830ce49e31490f686af3123732f5d.slice/crio-ee7d445115d11cd4904757ab54c668b00ca793e8a3b4c812de980c1069378cdd WatchSource:0}: Error finding container ee7d445115d11cd4904757ab54c668b00ca793e8a3b4c812de980c1069378cdd: Status 404 returned error can't find the container with id ee7d445115d11cd4904757ab54c668b00ca793e8a3b4c812de980c1069378cdd
Sep 11 11:48:00.059942 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.059744    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 11:42:59 +0000 UTC" deadline="2028-06-27 20:51:06.62088257 +0000 UTC"
Sep 11 11:48:00.059942 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.059809    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15729h3m6.561077926s"
Sep 11 11:48:00.070295 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.070235    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-220.ec2.internal" event={"ID":"52c830ce49e31490f686af3123732f5d","Type":"ContainerStarted","Data":"ee7d445115d11cd4904757ab54c668b00ca793e8a3b4c812de980c1069378cdd"}
Sep 11 11:48:00.072407 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.072375    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc" event={"ID":"c5ed4785-f3e8-424b-936d-588ebe7179a6","Type":"ContainerStarted","Data":"3bc0dbdd2c8fe113f34f84fc671e9505cdfeafc51cb64fa701f6791625c52674"}
Sep 11 11:48:00.073862 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.073799    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8" event={"ID":"72619839-fc9b-42d3-8442-d14c24b423f5","Type":"ContainerStarted","Data":"a41439e6755c9a27fce602e326d46332ce5fdb0cd87b8570426b792aebcbbbc2"}
Sep 11 11:48:00.076006 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.075841    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7q5t4" event={"ID":"fc1ebadd-d2ed-4de2-8225-21f60ffae756","Type":"ContainerStarted","Data":"0770d2e0e6a6bc7f29c8b2eb0161fe24431205353a4624749f85cdced7ed28e4"}
Sep 11 11:48:00.077682 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.077656    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fjbdx" event={"ID":"96f46a0c-d972-4490-a888-ca0f79174b50","Type":"ContainerStarted","Data":"2fff8442c82f860bbe92653ed6a1813edd31b7352958cbc8803b8222e0ea23ed"}
Sep 11 11:48:00.079572 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.079542    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vk54k" event={"ID":"714f5e6f-c22d-48de-b545-6be2e58968b0","Type":"ContainerStarted","Data":"7b70ab6e37aebf84721db0e69393e71f230caa93ad047ca3a3420445219b987d"}
Sep 11 11:48:00.081752 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.081729    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g7699" event={"ID":"25ce4b39-e461-46a5-be7e-e60fa1dad8f0","Type":"ContainerStarted","Data":"71d890d58242780a35a1e0f99b92c4dd677ecb0d68d0217d57c13bebe7a8df91"}
Sep 11 11:48:00.083728 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.083702    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f792v" event={"ID":"235f8771-f39a-471a-b452-2f62c27e0ccb","Type":"ContainerStarted","Data":"264029e9acf40d5ab141f721474e6d43122247c6347d9ee6d86307ba6b915c01"}
Sep 11 11:48:00.085351 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.085326    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-psw94" event={"ID":"c75825cf-1845-48bc-9e52-4c80394fc360","Type":"ContainerStarted","Data":"993589a38ca72a0df0507c199ec93296d92292dac97ab3d20fcceec984ca39c3"}
Sep 11 11:48:00.086775 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.086751    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vkfwj" event={"ID":"8bb6ad09-b9bd-41ef-bf5d-2ae5a3c28aff","Type":"ContainerStarted","Data":"f2b107284ef2738bf14ce1256ac5ba186d6cd7754db80348eae0b70ae600656e"}
Sep 11 11:48:00.089221 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.089195    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal" event={"ID":"4772d2a6d83dd9e80eabcadfa4ef7e19","Type":"ContainerStarted","Data":"4d10b1c2aabace40e4ebfd6cfae4258eca42462169e5bd53745a5dc0db086715"}
Sep 11 11:48:00.359969 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.359889    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 11:48:00.538670 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.538640    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 11:48:00.724296 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.724202    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs\") pod \"network-metrics-daemon-2xwtg\" (UID: \"ca51005b-bb90-4eac-8bc7-c64bb5463c3f\") " pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:00.724296 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.724253    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret\") pod \"global-pull-secret-syncer-sm98f\" (UID: \"584f8d60-a99b-4ab3-acbe-c5de6343284f\") " pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:00.724515 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:00.724386    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 11:48:00.724515 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:00.724459    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret podName:584f8d60-a99b-4ab3-acbe-c5de6343284f nodeName:}" failed. No retries permitted until 2026-09-11 11:48:02.724441304 +0000 UTC m=+5.146309680 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret") pod "global-pull-secret-syncer-sm98f" (UID: "584f8d60-a99b-4ab3-acbe-c5de6343284f") : object "kube-system"/"original-pull-secret" not registered
Sep 11 11:48:00.724922 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:00.724900    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 11:48:00.725041 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:00.724960    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs podName:ca51005b-bb90-4eac-8bc7-c64bb5463c3f nodeName:}" failed. No retries permitted until 2026-09-11 11:48:02.72494219 +0000 UTC m=+5.146810556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs") pod "network-metrics-daemon-2xwtg" (UID: "ca51005b-bb90-4eac-8bc7-c64bb5463c3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 11:48:00.825544 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:00.825243    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvzv\" (UniqueName: \"kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv\") pod \"network-check-target-p72gz\" (UID: \"83d5cf45-f64b-4e9d-8d77-088b52928c66\") " pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:00.825544 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:00.825399    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 11:48:00.825544 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:00.825424    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 11:48:00.825544 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:00.825438    2569 projected.go:196] Error preparing data for projected volume kube-api-access-9lvzv for pod openshift-network-diagnostics/network-check-target-p72gz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 11:48:00.825544 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:00.825495    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv podName:83d5cf45-f64b-4e9d-8d77-088b52928c66 nodeName:}" failed. No retries permitted until 2026-09-11 11:48:02.82547686 +0000 UTC m=+5.247345222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9lvzv" (UniqueName: "kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv") pod "network-check-target-p72gz" (UID: "83d5cf45-f64b-4e9d-8d77-088b52928c66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 11:48:01.060507 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:01.060119    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 11:42:59 +0000 UTC" deadline="2028-04-12 19:09:37.366872899 +0000 UTC"
Sep 11 11:48:01.060507 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:01.060161    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13903h21m36.30671679s"
Sep 11 11:48:01.063544 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:01.063517    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:01.063698 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:01.063676    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:48:01.063810 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:01.063768    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:01.063868 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:01.063850    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm98f" podUID="584f8d60-a99b-4ab3-acbe-c5de6343284f"
Sep 11 11:48:01.063929 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:01.063904    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:01.063984 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:01.063967    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:48:02.746157 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:02.745290    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret\") pod \"global-pull-secret-syncer-sm98f\" (UID: \"584f8d60-a99b-4ab3-acbe-c5de6343284f\") " pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:02.746157 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:02.745395    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs\") pod \"network-metrics-daemon-2xwtg\" (UID: \"ca51005b-bb90-4eac-8bc7-c64bb5463c3f\") " pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:02.746157 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:02.745512    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 11:48:02.746157 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:02.745574    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs podName:ca51005b-bb90-4eac-8bc7-c64bb5463c3f nodeName:}" failed. No retries permitted until 2026-09-11 11:48:06.745553588 +0000 UTC m=+9.167421948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs") pod "network-metrics-daemon-2xwtg" (UID: "ca51005b-bb90-4eac-8bc7-c64bb5463c3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 11:48:02.746157 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:02.746041    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 11:48:02.746157 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:02.746095    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret podName:584f8d60-a99b-4ab3-acbe-c5de6343284f nodeName:}" failed. No retries permitted until 2026-09-11 11:48:06.746079091 +0000 UTC m=+9.167947455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret") pod "global-pull-secret-syncer-sm98f" (UID: "584f8d60-a99b-4ab3-acbe-c5de6343284f") : object "kube-system"/"original-pull-secret" not registered
Sep 11 11:48:02.846615 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:02.846010    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvzv\" (UniqueName: \"kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv\") pod \"network-check-target-p72gz\" (UID: \"83d5cf45-f64b-4e9d-8d77-088b52928c66\") " pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:02.846615 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:02.846200    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 11:48:02.846615 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:02.846222    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 11:48:02.846615 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:02.846237    2569 projected.go:196] Error preparing data for projected volume kube-api-access-9lvzv for pod openshift-network-diagnostics/network-check-target-p72gz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 11:48:02.846615 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:02.846295    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv podName:83d5cf45-f64b-4e9d-8d77-088b52928c66 nodeName:}" failed. No retries permitted until 2026-09-11 11:48:06.846276677 +0000 UTC m=+9.268145040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9lvzv" (UniqueName: "kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv") pod "network-check-target-p72gz" (UID: "83d5cf45-f64b-4e9d-8d77-088b52928c66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 11:48:03.064490 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:03.063644    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:03.064490 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:03.063785    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:48:03.064490 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:03.064151    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:03.064490 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:03.064246    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm98f" podUID="584f8d60-a99b-4ab3-acbe-c5de6343284f"
Sep 11 11:48:03.064490 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:03.064308    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:03.064490 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:03.064375    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:48:05.063700 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:05.063650    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:05.064312 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:05.063666    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:05.064312 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:05.063790    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:48:05.064312 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:05.063816    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:05.064312 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:05.063918    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm98f" podUID="584f8d60-a99b-4ab3-acbe-c5de6343284f"
Sep 11 11:48:05.064312 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:05.064003    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:48:06.777650 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:06.777614    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret\") pod \"global-pull-secret-syncer-sm98f\" (UID: \"584f8d60-a99b-4ab3-acbe-c5de6343284f\") " pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:06.778115 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:06.777696    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs\") pod \"network-metrics-daemon-2xwtg\" (UID: \"ca51005b-bb90-4eac-8bc7-c64bb5463c3f\") " pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:06.778115 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:06.777783    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 11:48:06.778115 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:06.777843    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs podName:ca51005b-bb90-4eac-8bc7-c64bb5463c3f nodeName:}" failed. No retries permitted until 2026-09-11 11:48:14.777822711 +0000 UTC m=+17.199691081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs") pod "network-metrics-daemon-2xwtg" (UID: "ca51005b-bb90-4eac-8bc7-c64bb5463c3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 11:48:06.778115 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:06.777783    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 11:48:06.778115 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:06.777907    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret podName:584f8d60-a99b-4ab3-acbe-c5de6343284f nodeName:}" failed. No retries permitted until 2026-09-11 11:48:14.777893391 +0000 UTC m=+17.199761763 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret") pod "global-pull-secret-syncer-sm98f" (UID: "584f8d60-a99b-4ab3-acbe-c5de6343284f") : object "kube-system"/"original-pull-secret" not registered
Sep 11 11:48:06.878361 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:06.878208    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvzv\" (UniqueName: \"kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv\") pod \"network-check-target-p72gz\" (UID: \"83d5cf45-f64b-4e9d-8d77-088b52928c66\") " pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:06.878500 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:06.878416    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 11:48:06.878500 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:06.878441    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 11:48:06.878500 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:06.878454    2569 projected.go:196] Error preparing data for projected volume kube-api-access-9lvzv for pod openshift-network-diagnostics/network-check-target-p72gz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 11:48:06.878670 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:06.878513    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv podName:83d5cf45-f64b-4e9d-8d77-088b52928c66 nodeName:}" failed. No retries permitted until 2026-09-11 11:48:14.878495225 +0000 UTC m=+17.300363631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9lvzv" (UniqueName: "kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv") pod "network-check-target-p72gz" (UID: "83d5cf45-f64b-4e9d-8d77-088b52928c66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 11:48:07.064318 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:07.064271    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:07.064318 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:07.064294    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:07.064318 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:07.064297    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:07.064580 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:07.064400    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:48:07.064580 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:07.064471    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm98f" podUID="584f8d60-a99b-4ab3-acbe-c5de6343284f"
Sep 11 11:48:07.064580 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:07.064550    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:48:09.064467 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:09.063936    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:09.064467 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:09.064069    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:48:09.064467 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:09.064144    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:09.064467 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:09.064245    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm98f" podUID="584f8d60-a99b-4ab3-acbe-c5de6343284f"
Sep 11 11:48:09.064467 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:09.064306    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:09.064467 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:09.064369    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:48:11.063476 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:11.063445    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:11.063814 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:11.063446    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:11.063814 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:11.063547    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm98f" podUID="584f8d60-a99b-4ab3-acbe-c5de6343284f"
Sep 11 11:48:11.063814 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:11.063630    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:48:11.063814 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:11.063446    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:11.063814 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:11.063715    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:48:13.063652 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:13.063564    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:13.063652 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:13.063624    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:13.064274 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:13.063698    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:13.064274 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:13.063703    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm98f" podUID="584f8d60-a99b-4ab3-acbe-c5de6343284f"
Sep 11 11:48:13.064274 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:13.063805    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:48:13.064274 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:13.063897    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:48:14.837188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:14.837155    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret\") pod \"global-pull-secret-syncer-sm98f\" (UID: \"584f8d60-a99b-4ab3-acbe-c5de6343284f\") " pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:14.837640 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:14.837323    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 11:48:14.837640 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:14.837388    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret podName:584f8d60-a99b-4ab3-acbe-c5de6343284f nodeName:}" failed. No retries permitted until 2026-09-11 11:48:30.837367102 +0000 UTC m=+33.259235483 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret") pod "global-pull-secret-syncer-sm98f" (UID: "584f8d60-a99b-4ab3-acbe-c5de6343284f") : object "kube-system"/"original-pull-secret" not registered
Sep 11 11:48:14.837640 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:14.837323    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs\") pod \"network-metrics-daemon-2xwtg\" (UID: \"ca51005b-bb90-4eac-8bc7-c64bb5463c3f\") " pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:14.837640 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:14.837409    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 11:48:14.837640 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:14.837470    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs podName:ca51005b-bb90-4eac-8bc7-c64bb5463c3f nodeName:}" failed. No retries permitted until 2026-09-11 11:48:30.837452893 +0000 UTC m=+33.259321257 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs") pod "network-metrics-daemon-2xwtg" (UID: "ca51005b-bb90-4eac-8bc7-c64bb5463c3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 11:48:14.937822 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:14.937791    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvzv\" (UniqueName: \"kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv\") pod \"network-check-target-p72gz\" (UID: \"83d5cf45-f64b-4e9d-8d77-088b52928c66\") " pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:14.937981 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:14.937944    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 11:48:14.937981 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:14.937960    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 11:48:14.937981 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:14.937969    2569 projected.go:196] Error preparing data for projected volume kube-api-access-9lvzv for pod openshift-network-diagnostics/network-check-target-p72gz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 11:48:14.938148 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:14.938017    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv podName:83d5cf45-f64b-4e9d-8d77-088b52928c66 nodeName:}" failed. No retries permitted until 2026-09-11 11:48:30.938003468 +0000 UTC m=+33.359871840 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9lvzv" (UniqueName: "kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv") pod "network-check-target-p72gz" (UID: "83d5cf45-f64b-4e9d-8d77-088b52928c66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 11:48:15.064123 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:15.064090    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:15.064288 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:15.064193    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:15.064288 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:15.064206    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm98f" podUID="584f8d60-a99b-4ab3-acbe-c5de6343284f"
Sep 11 11:48:15.064378 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:15.064301    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:48:15.064378 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:15.064350    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:15.064499 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:15.064435    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:48:17.064311 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:17.064258    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:17.064788 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:17.064377    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:17.064788 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:17.064402    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm98f" podUID="584f8d60-a99b-4ab3-acbe-c5de6343284f"
Sep 11 11:48:17.064788 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:17.064414    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:17.064788 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:17.064518    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:48:17.064788 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:17.064612    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:48:19.063478 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:19.063450    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:19.063856 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:19.063498    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:19.063856 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:19.063653    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:19.063856 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:19.063667    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:48:19.063856 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:19.063735    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm98f" podUID="584f8d60-a99b-4ab3-acbe-c5de6343284f"
Sep 11 11:48:19.063856 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:19.063792    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:48:19.145975 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:19.145936    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-220.ec2.internal" event={"ID":"52c830ce49e31490f686af3123732f5d","Type":"ContainerStarted","Data":"9dd8afd07ff7df4f557957bfbdb2dadcd7fd3d499b456f647ee2a247633df49b"}
Sep 11 11:48:19.149637 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:19.149616    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8" event={"ID":"72619839-fc9b-42d3-8442-d14c24b423f5","Type":"ContainerStarted","Data":"9330a52e15be475da7c6525b408f4e9deef5c075f5244dd907d2baeba1df23fa"}
Sep 11 11:48:19.151098 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:19.151076    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f792v" event={"ID":"235f8771-f39a-471a-b452-2f62c27e0ccb","Type":"ContainerStarted","Data":"52e41f60aa9f61087983f5f278188e9d57432d6ab48b20208af2e350e1deae9a"}
Sep 11 11:48:19.186957 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:19.186911    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-220.ec2.internal" podStartSLOduration=20.186899078 podStartE2EDuration="20.186899078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:48:19.166920652 +0000 UTC m=+21.588789034" watchObservedRunningTime="2026-09-11 11:48:19.186899078 +0000 UTC m=+21.608767479"
Sep 11 11:48:19.187181 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:19.187164    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f792v" podStartSLOduration=1.611294377 podStartE2EDuration="21.187159598s" totalImagesPullingTime="19.575865221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:48:19.186782976 +0000 UTC m=+21.608651366" watchObservedRunningTime="2026-09-11 11:48:19.187159598 +0000 UTC m=+21.609027978"
Sep 11 11:48:20.154022 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.153989    2569 generic.go:356] "Generic (PLEG): container finished" podID="4772d2a6d83dd9e80eabcadfa4ef7e19" containerID="99a4b957a7cad8ef6b2b178f61d62cac79ffdd6737419fd687aad76116be342e" exitCode=0
Sep 11 11:48:20.154668 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.154043    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal" event={"ID":"4772d2a6d83dd9e80eabcadfa4ef7e19","Type":"ContainerDied","Data":"99a4b957a7cad8ef6b2b178f61d62cac79ffdd6737419fd687aad76116be342e"}
Sep 11 11:48:20.155635 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.155609    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc" event={"ID":"c5ed4785-f3e8-424b-936d-588ebe7179a6","Type":"ContainerStarted","Data":"0560c6abfd33912e771f245ac803f9576adc5960ae67fec3b568961e0a671337"}
Sep 11 11:48:20.158253 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.158234    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 11:48:20.158566 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.158533    2569 generic.go:356] "Generic (PLEG): container finished" podID="72619839-fc9b-42d3-8442-d14c24b423f5" containerID="c5dddbf0e3804a7d60a2817697a1e6956bf67754c7d235eb1c8dd604320aacbc" exitCode=1
Sep 11 11:48:20.158670 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.158612    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8" event={"ID":"72619839-fc9b-42d3-8442-d14c24b423f5","Type":"ContainerStarted","Data":"b601d1f6a711ce04d1d88a47611cfec7ad8dbfe417fe02d51d7a6fc1d7af4cb2"}
Sep 11 11:48:20.158729 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.158664    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8" event={"ID":"72619839-fc9b-42d3-8442-d14c24b423f5","Type":"ContainerStarted","Data":"19ad49101e4470ab1b2e75569c26e84395853a0e5c1f2e6aea1a67f5b97acd13"}
Sep 11 11:48:20.158729 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.158721    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8" event={"ID":"72619839-fc9b-42d3-8442-d14c24b423f5","Type":"ContainerStarted","Data":"ffb356e65eb7f9f52de8837e87a7b6ddcdbf2b6c41a1a7d07e65d66eba468ff8"}
Sep 11 11:48:20.158810 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.158781    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8" event={"ID":"72619839-fc9b-42d3-8442-d14c24b423f5","Type":"ContainerStarted","Data":"da6c26336c64904f71922167cb077859ff9b549d7953316f6cb871a2c4e15278"}
Sep 11 11:48:20.158862 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.158823    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8" event={"ID":"72619839-fc9b-42d3-8442-d14c24b423f5","Type":"ContainerDied","Data":"c5dddbf0e3804a7d60a2817697a1e6956bf67754c7d235eb1c8dd604320aacbc"}
Sep 11 11:48:20.160057 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.160037    2569 generic.go:356] "Generic (PLEG): container finished" podID="fc1ebadd-d2ed-4de2-8225-21f60ffae756" containerID="4b44f9d80de0d9784bcae98f473f69194908b2f8bc316cb4581c8e157af6a925" exitCode=0
Sep 11 11:48:20.160154 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.160093    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7q5t4" event={"ID":"fc1ebadd-d2ed-4de2-8225-21f60ffae756","Type":"ContainerDied","Data":"4b44f9d80de0d9784bcae98f473f69194908b2f8bc316cb4581c8e157af6a925"}
Sep 11 11:48:20.161475 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.161410    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fjbdx" event={"ID":"96f46a0c-d972-4490-a888-ca0f79174b50","Type":"ContainerStarted","Data":"210fe872d8628a1abf7df6d7351eef6205077b012e81d6ae50e6a9b5a61c1af6"}
Sep 11 11:48:20.162964 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.162944    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vk54k" event={"ID":"714f5e6f-c22d-48de-b545-6be2e58968b0","Type":"ContainerStarted","Data":"d016d8c0e5b1298a47f9db6d1fa00678f6f578e9098c9b188b535318b8fb4a91"}
Sep 11 11:48:20.164385 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.164358    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-psw94" event={"ID":"c75825cf-1845-48bc-9e52-4c80394fc360","Type":"ContainerStarted","Data":"a9d2228ef94a15c07c441e72dc88fccd6fb958a8e81f07b1660e03f8207f2646"}
Sep 11 11:48:20.166277 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.166255    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vkfwj" event={"ID":"8bb6ad09-b9bd-41ef-bf5d-2ae5a3c28aff","Type":"ContainerStarted","Data":"8e8d892cddfc70d04729e60b1c1f8ffa3e31e41e119ed396c0078702ccb852ad"}
Sep 11 11:48:20.222944 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.222906    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vkfwj" podStartSLOduration=2.653350404 podStartE2EDuration="22.222892157s" totalImagesPullingTime="19.569541753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:48:20.222627479 +0000 UTC m=+22.644495875" watchObservedRunningTime="2026-09-11 11:48:20.222892157 +0000 UTC m=+22.644760537"
Sep 11 11:48:20.254405 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.254368    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fjbdx" podStartSLOduration=2.754299942 podStartE2EDuration="22.254355624s" totalImagesPullingTime="19.500055682s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:48:20.254256258 +0000 UTC m=+22.676124639" watchObservedRunningTime="2026-09-11 11:48:20.254355624 +0000 UTC m=+22.676224003"
Sep 11 11:48:20.254702 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.254680    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vk54k" podStartSLOduration=2.739639258 podStartE2EDuration="22.254673011s" totalImagesPullingTime="19.515033753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:48:20.23814534 +0000 UTC m=+22.660013719" watchObservedRunningTime="2026-09-11 11:48:20.254673011 +0000 UTC m=+22.676541391"
Sep 11 11:48:20.272503 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.272467    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-psw94" podStartSLOduration=2.437519552 podStartE2EDuration="22.27245469s" totalImagesPullingTime="19.834935138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:48:20.27208635 +0000 UTC m=+22.693954731" watchObservedRunningTime="2026-09-11 11:48:20.27245469 +0000 UTC m=+22.694323070"
Sep 11 11:48:20.435467 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.435415    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vkfwj"
Sep 11 11:48:20.435889 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:20.435873    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vkfwj"
Sep 11 11:48:21.035575 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:21.035388    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 11:48:21.064070 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:21.064044    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:21.064070 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:21.064069    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:21.064237 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:21.064052    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:21.064237 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:21.064148    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm98f" podUID="584f8d60-a99b-4ab3-acbe-c5de6343284f"
Sep 11 11:48:21.064237 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:21.064225    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:48:21.064398 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:21.064303    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:48:21.069306 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:21.069207    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T11:48:21.03557314Z","UUID":"3fe25198-9ed9-4ff4-83b1-e55cf0be76d7","Handler":null,"Name":"","Endpoint":""}
Sep 11 11:48:21.070423 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:21.070407    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 11:48:21.070526 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:21.070436    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 11:48:21.170763 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:21.170669    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal" event={"ID":"4772d2a6d83dd9e80eabcadfa4ef7e19","Type":"ContainerStarted","Data":"ece561c98476ee1a95b17497902be526534b75db155c05ea5c4b8a09c3b87fd9"}
Sep 11 11:48:21.172953 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:21.172921    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc" event={"ID":"c5ed4785-f3e8-424b-936d-588ebe7179a6","Type":"ContainerStarted","Data":"2f77b51ef938afb0c7718140503d0d0e428c2d5c0829a756097b11aa8fc4f71e"}
Sep 11 11:48:21.174650 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:21.174626    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g7699" event={"ID":"25ce4b39-e461-46a5-be7e-e60fa1dad8f0","Type":"ContainerStarted","Data":"4bd1c2afd0532a23c129ebab3ccf3d2676a898c22d0dfe59099111ee6a368428"}
Sep 11 11:48:21.174958 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:21.174914    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vkfwj"
Sep 11 11:48:21.175504 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:21.175474    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vkfwj"
Sep 11 11:48:21.187167 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:21.187120    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-220.ec2.internal" podStartSLOduration=22.187103852 podStartE2EDuration="22.187103852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:48:21.186862382 +0000 UTC m=+23.608730783" watchObservedRunningTime="2026-09-11 11:48:21.187103852 +0000 UTC m=+23.608972234"
Sep 11 11:48:21.216899 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:21.216858    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g7699" podStartSLOduration=3.678535703 podStartE2EDuration="23.216847081s" totalImagesPullingTime="19.538311378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:48:21.216613879 +0000 UTC m=+23.638482276" watchObservedRunningTime="2026-09-11 11:48:21.216847081 +0000 UTC m=+23.638715462"
Sep 11 11:48:22.180406 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:22.180379    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 11:48:22.180854 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:22.180804    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8" event={"ID":"72619839-fc9b-42d3-8442-d14c24b423f5","Type":"ContainerStarted","Data":"2c7b702ee4765bbf02235f83acbb2304f0f07a150e04524b126eaa3e01fd32c4"}
Sep 11 11:48:23.064130 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:23.064098    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:23.064130 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:23.064128    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:23.064362 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:23.064202    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:23.064362 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:23.064218    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:48:23.064362 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:23.064325    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:48:23.064493 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:23.064416    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm98f" podUID="584f8d60-a99b-4ab3-acbe-c5de6343284f"
Sep 11 11:48:23.185138 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:23.185104    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc" event={"ID":"c5ed4785-f3e8-424b-936d-588ebe7179a6","Type":"ContainerStarted","Data":"ae9d79f0a0412d5483f961c27ca870d9fc2088f4585461c4e9baf8b4cf3f2824"}
Sep 11 11:48:23.206008 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:23.205957    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4wvzc" podStartSLOduration=3.511571706 podStartE2EDuration="25.205938425s" totalImagesPullingTime="21.694366719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:47:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:48:23.205215953 +0000 UTC m=+25.627084354" watchObservedRunningTime="2026-09-11 11:48:23.205938425 +0000 UTC m=+25.627806806"
Sep 11 11:48:25.063707 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:25.063479    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:25.064141 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:25.063519    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:25.064141 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:25.063749    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:48:25.064141 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:25.063521    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:25.064141 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:25.063872    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:48:25.064141 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:25.063806    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm98f" podUID="584f8d60-a99b-4ab3-acbe-c5de6343284f"
Sep 11 11:48:25.192914 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:25.192893    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 11:48:25.193310 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:25.193292    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8" event={"ID":"72619839-fc9b-42d3-8442-d14c24b423f5","Type":"ContainerStarted","Data":"6a3abe22eaa51b5293948f94d0a25c1207eedaf1592b2855ffb7d7796aa8eb31"}
Sep 11 11:48:25.193579 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:25.193549    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:48:25.193766 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:25.193623    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:48:25.193766 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:25.193666    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:48:25.193766 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:25.193694    2569 scope.go:117] "RemoveContainer" containerID="c5dddbf0e3804a7d60a2817697a1e6956bf67754c7d235eb1c8dd604320aacbc"
Sep 11 11:48:25.196004 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:25.195983    2569 generic.go:356] "Generic (PLEG): container finished" podID="fc1ebadd-d2ed-4de2-8225-21f60ffae756" containerID="92b89cf8205ce56f926cc41806480d41044e42bc70b680c3a364112a55d8e804" exitCode=0
Sep 11 11:48:25.196112 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:25.196024    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7q5t4" event={"ID":"fc1ebadd-d2ed-4de2-8225-21f60ffae756","Type":"ContainerDied","Data":"92b89cf8205ce56f926cc41806480d41044e42bc70b680c3a364112a55d8e804"}
Sep 11 11:48:25.209387 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:25.209368    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:48:25.209510 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:25.209497    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:48:26.203478 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:26.203442    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 11:48:26.204112 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:26.203914    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8" event={"ID":"72619839-fc9b-42d3-8442-d14c24b423f5","Type":"ContainerStarted","Data":"d3a48a94abf74ebb1002fb7d6b62b0b0c3b4aa0bee9a3a792d0f95c7740a7c38"}
Sep 11 11:48:26.600792 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:26.600624    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8" podStartSLOduration=9.053771864 podStartE2EDuration="28.600609665s" totalImagesPullingTime="19.546837801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:48:26.237395702 +0000 UTC m=+28.659264108" watchObservedRunningTime="2026-09-11 11:48:26.600609665 +0000 UTC m=+29.022478041"
Sep 11 11:48:26.601147 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:26.601135    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p72gz"]
Sep 11 11:48:26.601262 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:26.601250    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:26.601340 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:26.601325    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:48:26.613232 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:26.613210    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2xwtg"]
Sep 11 11:48:26.613339 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:26.613325    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:26.613473 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:26.613442    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:48:26.619597 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:26.619565    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sm98f"]
Sep 11 11:48:26.619689 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:26.619667    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:26.619802 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:26.619758    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm98f" podUID="584f8d60-a99b-4ab3-acbe-c5de6343284f"
Sep 11 11:48:27.208538 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:27.208499    2569 generic.go:356] "Generic (PLEG): container finished" podID="fc1ebadd-d2ed-4de2-8225-21f60ffae756" containerID="1a210dd2aceb83ed14284a92b2ce5549d3d2a5461cfc911857fc999fd8f40316" exitCode=0
Sep 11 11:48:27.208960 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:27.208607    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7q5t4" event={"ID":"fc1ebadd-d2ed-4de2-8225-21f60ffae756","Type":"ContainerDied","Data":"1a210dd2aceb83ed14284a92b2ce5549d3d2a5461cfc911857fc999fd8f40316"}
Sep 11 11:48:28.064179 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:28.064152    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:28.064282 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:28.064212    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:28.064342 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:28.064273    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:48:28.064425 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:28.064397    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:48:28.216926 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:28.216846    2569 generic.go:356] "Generic (PLEG): container finished" podID="fc1ebadd-d2ed-4de2-8225-21f60ffae756" containerID="9438730eba9c29edeb999a6795a21cb7a2d795b70e9539b99109bc0a72d28a0f" exitCode=0
Sep 11 11:48:28.216926 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:28.216893    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7q5t4" event={"ID":"fc1ebadd-d2ed-4de2-8225-21f60ffae756","Type":"ContainerDied","Data":"9438730eba9c29edeb999a6795a21cb7a2d795b70e9539b99109bc0a72d28a0f"}
Sep 11 11:48:29.063269 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:29.063172    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:29.063440 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:29.063299    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm98f" podUID="584f8d60-a99b-4ab3-acbe-c5de6343284f"
Sep 11 11:48:30.063523 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:30.063489    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:30.064044 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:30.063520    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:30.064044 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:30.063631    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:48:30.064044 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:30.063786    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:48:30.854967 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:30.854931    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs\") pod \"network-metrics-daemon-2xwtg\" (UID: \"ca51005b-bb90-4eac-8bc7-c64bb5463c3f\") " pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:30.855168 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:30.854981    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret\") pod \"global-pull-secret-syncer-sm98f\" (UID: \"584f8d60-a99b-4ab3-acbe-c5de6343284f\") " pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:30.855168 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:30.855101    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 11:48:30.855168 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:30.855103    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 11:48:30.855168 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:30.855165    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret podName:584f8d60-a99b-4ab3-acbe-c5de6343284f nodeName:}" failed. No retries permitted until 2026-09-11 11:49:02.85514825 +0000 UTC m=+65.277016612 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret") pod "global-pull-secret-syncer-sm98f" (UID: "584f8d60-a99b-4ab3-acbe-c5de6343284f") : object "kube-system"/"original-pull-secret" not registered
Sep 11 11:48:30.855330 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:30.855184    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs podName:ca51005b-bb90-4eac-8bc7-c64bb5463c3f nodeName:}" failed. No retries permitted until 2026-09-11 11:49:02.855174736 +0000 UTC m=+65.277043099 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs") pod "network-metrics-daemon-2xwtg" (UID: "ca51005b-bb90-4eac-8bc7-c64bb5463c3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 11:48:30.955737 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:30.955705    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvzv\" (UniqueName: \"kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv\") pod \"network-check-target-p72gz\" (UID: \"83d5cf45-f64b-4e9d-8d77-088b52928c66\") " pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:30.955900 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:30.955861    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 11:48:30.955900 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:30.955881    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 11:48:30.955900 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:30.955893    2569 projected.go:196] Error preparing data for projected volume kube-api-access-9lvzv for pod openshift-network-diagnostics/network-check-target-p72gz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 11:48:30.956025 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:30.955949    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv podName:83d5cf45-f64b-4e9d-8d77-088b52928c66 nodeName:}" failed. No retries permitted until 2026-09-11 11:49:02.955931977 +0000 UTC m=+65.377800349 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9lvzv" (UniqueName: "kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv") pod "network-check-target-p72gz" (UID: "83d5cf45-f64b-4e9d-8d77-088b52928c66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 11:48:31.064254 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:31.064226    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:31.064795 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:31.064335    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm98f" podUID="584f8d60-a99b-4ab3-acbe-c5de6343284f"
Sep 11 11:48:32.064191 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.064159    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:32.064352 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.064168    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:32.064352 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:32.064304    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:48:32.064740 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:32.064359    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p72gz" podUID="83d5cf45-f64b-4e9d-8d77-088b52928c66"
Sep 11 11:48:32.416651 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.416555    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-220.ec2.internal" event="NodeReady"
Sep 11 11:48:32.416806 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.416703    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 11:48:32.464789 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.464753    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hmj9n"]
Sep 11 11:48:32.491213 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.491183    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-c4qfd"]
Sep 11 11:48:32.491385 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.491338    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:48:32.494135 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.494112    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 11:48:32.494554 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.494537    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 11:48:32.494554 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.494547    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m9sw5\""
Sep 11 11:48:32.513056 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.513032    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hmj9n"]
Sep 11 11:48:32.513170 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.513072    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c4qfd"]
Sep 11 11:48:32.513170 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.513156    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c4qfd"
Sep 11 11:48:32.517876 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.517644    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 11:48:32.517876 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.517730    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wpl6b\""
Sep 11 11:48:32.518262 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.518027    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 11:48:32.518803 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.518786    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 11:48:32.566572 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.566542    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:48:32.566737 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.566581    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c76d6c-b4b3-442a-8081-0ec81e5894a6-tmp-dir\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:48:32.566737 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.566708    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgdkc\" (UniqueName: \"kubernetes.io/projected/89c76d6c-b4b3-442a-8081-0ec81e5894a6-kube-api-access-wgdkc\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:48:32.566835 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.566753    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89c76d6c-b4b3-442a-8081-0ec81e5894a6-config-volume\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:48:32.667948 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.667875    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/775678cb-ae06-458a-a9f1-1ab4b5f48928-cert\") pod \"ingress-canary-c4qfd\" (UID: \"775678cb-ae06-458a-a9f1-1ab4b5f48928\") " pod="openshift-ingress-canary/ingress-canary-c4qfd"
Sep 11 11:48:32.667948 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.667916    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89c76d6c-b4b3-442a-8081-0ec81e5894a6-config-volume\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:48:32.668152 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.667954    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:48:32.668152 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.667976    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c76d6c-b4b3-442a-8081-0ec81e5894a6-tmp-dir\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:48:32.668152 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:32.668087    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 11:48:32.668281 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:32.668157    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls podName:89c76d6c-b4b3-442a-8081-0ec81e5894a6 nodeName:}" failed. No retries permitted until 2026-09-11 11:48:33.168135446 +0000 UTC m=+35.590003812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls") pod "dns-default-hmj9n" (UID: "89c76d6c-b4b3-442a-8081-0ec81e5894a6") : secret "dns-default-metrics-tls" not found
Sep 11 11:48:32.668281 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.668180    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgdkc\" (UniqueName: \"kubernetes.io/projected/89c76d6c-b4b3-442a-8081-0ec81e5894a6-kube-api-access-wgdkc\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:48:32.668281 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.668213    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq8vm\" (UniqueName: \"kubernetes.io/projected/775678cb-ae06-458a-a9f1-1ab4b5f48928-kube-api-access-kq8vm\") pod \"ingress-canary-c4qfd\" (UID: \"775678cb-ae06-458a-a9f1-1ab4b5f48928\") " pod="openshift-ingress-canary/ingress-canary-c4qfd"
Sep 11 11:48:32.668424 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.668338    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c76d6c-b4b3-442a-8081-0ec81e5894a6-tmp-dir\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:48:32.668562 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.668543    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89c76d6c-b4b3-442a-8081-0ec81e5894a6-config-volume\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:48:32.682021 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.681999    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgdkc\" (UniqueName: \"kubernetes.io/projected/89c76d6c-b4b3-442a-8081-0ec81e5894a6-kube-api-access-wgdkc\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:48:32.769503 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.769479    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq8vm\" (UniqueName: \"kubernetes.io/projected/775678cb-ae06-458a-a9f1-1ab4b5f48928-kube-api-access-kq8vm\") pod \"ingress-canary-c4qfd\" (UID: \"775678cb-ae06-458a-a9f1-1ab4b5f48928\") " pod="openshift-ingress-canary/ingress-canary-c4qfd"
Sep 11 11:48:32.769630 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.769510    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/775678cb-ae06-458a-a9f1-1ab4b5f48928-cert\") pod \"ingress-canary-c4qfd\" (UID: \"775678cb-ae06-458a-a9f1-1ab4b5f48928\") " pod="openshift-ingress-canary/ingress-canary-c4qfd"
Sep 11 11:48:32.771893 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.771874    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/775678cb-ae06-458a-a9f1-1ab4b5f48928-cert\") pod \"ingress-canary-c4qfd\" (UID: \"775678cb-ae06-458a-a9f1-1ab4b5f48928\") " pod="openshift-ingress-canary/ingress-canary-c4qfd"
Sep 11 11:48:32.778736 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.778714    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq8vm\" (UniqueName: \"kubernetes.io/projected/775678cb-ae06-458a-a9f1-1ab4b5f48928-kube-api-access-kq8vm\") pod \"ingress-canary-c4qfd\" (UID: \"775678cb-ae06-458a-a9f1-1ab4b5f48928\") " pod="openshift-ingress-canary/ingress-canary-c4qfd"
Sep 11 11:48:32.823803 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:32.823783    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c4qfd"
Sep 11 11:48:33.063798 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:33.063461    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:48:33.066166 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:33.066140    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 11:48:33.122893 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:33.122859    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c4qfd"]
Sep 11 11:48:33.135445 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:48:33.135420    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775678cb_ae06_458a_a9f1_1ab4b5f48928.slice/crio-2ba126c40b68e2b0b94789c5edf9c34bbafda8539b5c78eba21ff9f29c63c2ba WatchSource:0}: Error finding container 2ba126c40b68e2b0b94789c5edf9c34bbafda8539b5c78eba21ff9f29c63c2ba: Status 404 returned error can't find the container with id 2ba126c40b68e2b0b94789c5edf9c34bbafda8539b5c78eba21ff9f29c63c2ba
Sep 11 11:48:33.172226 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:33.172201    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:48:33.172351 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:33.172307    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 11:48:33.172351 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:33.172350    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls podName:89c76d6c-b4b3-442a-8081-0ec81e5894a6 nodeName:}" failed. No retries permitted until 2026-09-11 11:48:34.172337226 +0000 UTC m=+36.594205597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls") pod "dns-default-hmj9n" (UID: "89c76d6c-b4b3-442a-8081-0ec81e5894a6") : secret "dns-default-metrics-tls" not found
Sep 11 11:48:33.229278 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:33.229249    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c4qfd" event={"ID":"775678cb-ae06-458a-a9f1-1ab4b5f48928","Type":"ContainerStarted","Data":"2ba126c40b68e2b0b94789c5edf9c34bbafda8539b5c78eba21ff9f29c63c2ba"}
Sep 11 11:48:33.232225 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:33.232200    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7q5t4" event={"ID":"fc1ebadd-d2ed-4de2-8225-21f60ffae756","Type":"ContainerStarted","Data":"3eee4b61707183bc7f6a3ab70bf4f301b3ea230500fb41ef3dc6a69e33af4403"}
Sep 11 11:48:34.067662 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:34.067554    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:48:34.068258 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:34.067610    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:48:34.071790 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:34.071722    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 11:48:34.071790 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:34.071736    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 11:48:34.071976 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:34.071799    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5sht6\""
Sep 11 11:48:34.071976 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:34.071822    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xblbl\""
Sep 11 11:48:34.071976 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:34.071848    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 11:48:34.177597 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:34.177558    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:48:34.177779 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:34.177758    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 11:48:34.177894 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:34.177822    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls podName:89c76d6c-b4b3-442a-8081-0ec81e5894a6 nodeName:}" failed. No retries permitted until 2026-09-11 11:48:36.177802996 +0000 UTC m=+38.599671361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls") pod "dns-default-hmj9n" (UID: "89c76d6c-b4b3-442a-8081-0ec81e5894a6") : secret "dns-default-metrics-tls" not found
Sep 11 11:48:34.237676 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:34.237646    2569 generic.go:356] "Generic (PLEG): container finished" podID="fc1ebadd-d2ed-4de2-8225-21f60ffae756" containerID="3eee4b61707183bc7f6a3ab70bf4f301b3ea230500fb41ef3dc6a69e33af4403" exitCode=0
Sep 11 11:48:34.237790 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:34.237690    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7q5t4" event={"ID":"fc1ebadd-d2ed-4de2-8225-21f60ffae756","Type":"ContainerDied","Data":"3eee4b61707183bc7f6a3ab70bf4f301b3ea230500fb41ef3dc6a69e33af4403"}
Sep 11 11:48:35.244206 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:35.244175    2569 generic.go:356] "Generic (PLEG): container finished" podID="fc1ebadd-d2ed-4de2-8225-21f60ffae756" containerID="5c2cf28c364dcbab073fff011ea5e528e60a0926dbcf45d1431026ffe683e85a" exitCode=0
Sep 11 11:48:35.244789 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:35.244226    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7q5t4" event={"ID":"fc1ebadd-d2ed-4de2-8225-21f60ffae756","Type":"ContainerDied","Data":"5c2cf28c364dcbab073fff011ea5e528e60a0926dbcf45d1431026ffe683e85a"}
Sep 11 11:48:36.193222 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:36.193049    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:48:36.193354 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:36.193193    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 11:48:36.193354 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:36.193286    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls podName:89c76d6c-b4b3-442a-8081-0ec81e5894a6 nodeName:}" failed. No retries permitted until 2026-09-11 11:48:40.193271785 +0000 UTC m=+42.615140157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls") pod "dns-default-hmj9n" (UID: "89c76d6c-b4b3-442a-8081-0ec81e5894a6") : secret "dns-default-metrics-tls" not found
Sep 11 11:48:36.250112 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:36.250079    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7q5t4" event={"ID":"fc1ebadd-d2ed-4de2-8225-21f60ffae756","Type":"ContainerStarted","Data":"f7bb7acc296eb0f5afc913758f25e1c8f8c3f830c5d22fff10ab15ab919b5a72"}
Sep 11 11:48:36.251528 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:36.251499    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c4qfd" event={"ID":"775678cb-ae06-458a-a9f1-1ab4b5f48928","Type":"ContainerStarted","Data":"d0067923213384826c993d8fde74e571a25d1821b151b31e4c09462434cee16b"}
Sep 11 11:48:36.275166 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:36.275106    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7q5t4" podStartSLOduration=7.363006154 podStartE2EDuration="38.275093399s" totalImagesPullingTime="30.912087245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:47:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:48:36.273996276 +0000 UTC m=+38.695864664" watchObservedRunningTime="2026-09-11 11:48:36.275093399 +0000 UTC m=+38.696961844"
Sep 11 11:48:36.292137 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:36.292099    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-c4qfd" podStartSLOduration=1.802180489 podStartE2EDuration="4.292087013s" totalImagesPullingTime="2.489906524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:48:36.291746061 +0000 UTC m=+38.713614467" watchObservedRunningTime="2026-09-11 11:48:36.292087013 +0000 UTC m=+38.713955393"
Sep 11 11:48:40.219449 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:40.219412    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:48:40.219845 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:40.219575    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 11:48:40.219845 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:40.219665    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls podName:89c76d6c-b4b3-442a-8081-0ec81e5894a6 nodeName:}" failed. No retries permitted until 2026-09-11 11:48:48.219645428 +0000 UTC m=+50.641513793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls") pod "dns-default-hmj9n" (UID: "89c76d6c-b4b3-442a-8081-0ec81e5894a6") : secret "dns-default-metrics-tls" not found
Sep 11 11:48:48.270993 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:48.270957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:48:48.271439 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:48.271089    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 11:48:48.271439 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:48:48.271146    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls podName:89c76d6c-b4b3-442a-8081-0ec81e5894a6 nodeName:}" failed. No retries permitted until 2026-09-11 11:49:04.271130284 +0000 UTC m=+66.692998642 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls") pod "dns-default-hmj9n" (UID: "89c76d6c-b4b3-442a-8081-0ec81e5894a6") : secret "dns-default-metrics-tls" not found
Sep 11 11:48:57.220399 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:48:57.220367    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b6xv8"
Sep 11 11:49:02.868854 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:02.868817    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs\") pod \"network-metrics-daemon-2xwtg\" (UID: \"ca51005b-bb90-4eac-8bc7-c64bb5463c3f\") " pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:49:02.868854 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:02.868858    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret\") pod \"global-pull-secret-syncer-sm98f\" (UID: \"584f8d60-a99b-4ab3-acbe-c5de6343284f\") " pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:49:02.871288 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:02.871264    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 11:49:02.871445 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:02.871373    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 11:49:02.879258 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:49:02.879232    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 11:49:02.879337 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:49:02.879294    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs podName:ca51005b-bb90-4eac-8bc7-c64bb5463c3f nodeName:}" failed. No retries permitted until 2026-09-11 11:50:06.879277526 +0000 UTC m=+129.301145889 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs") pod "network-metrics-daemon-2xwtg" (UID: "ca51005b-bb90-4eac-8bc7-c64bb5463c3f") : secret "metrics-daemon-secret" not found
Sep 11 11:49:02.882077 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:02.882057    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/584f8d60-a99b-4ab3-acbe-c5de6343284f-original-pull-secret\") pod \"global-pull-secret-syncer-sm98f\" (UID: \"584f8d60-a99b-4ab3-acbe-c5de6343284f\") " pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:49:02.969740 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:02.969711    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvzv\" (UniqueName: \"kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv\") pod \"network-check-target-p72gz\" (UID: \"83d5cf45-f64b-4e9d-8d77-088b52928c66\") " pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:49:02.972510 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:02.972493    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 11:49:02.982684 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:02.982665    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 11:49:02.993657 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:02.993635    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lvzv\" (UniqueName: \"kubernetes.io/projected/83d5cf45-f64b-4e9d-8d77-088b52928c66-kube-api-access-9lvzv\") pod \"network-check-target-p72gz\" (UID: \"83d5cf45-f64b-4e9d-8d77-088b52928c66\") " pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:49:03.072741 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:03.072716    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm98f"
Sep 11 11:49:03.191788 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:03.191768    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xblbl\""
Sep 11 11:49:03.196262 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:03.196246    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:49:03.372444 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:03.372375    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sm98f"]
Sep 11 11:49:03.375869 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:49:03.375843    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584f8d60_a99b_4ab3_acbe_c5de6343284f.slice/crio-cbee99c7775e042f268e515d3cffe18cd4c2a625a4af85467ec789cdb4594372 WatchSource:0}: Error finding container cbee99c7775e042f268e515d3cffe18cd4c2a625a4af85467ec789cdb4594372: Status 404 returned error can't find the container with id cbee99c7775e042f268e515d3cffe18cd4c2a625a4af85467ec789cdb4594372
Sep 11 11:49:03.478052 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:03.477999    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p72gz"]
Sep 11 11:49:03.480479 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:49:03.480453    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d5cf45_f64b_4e9d_8d77_088b52928c66.slice/crio-b6721b2b27f4f6916e50dacfb7c3c48c6f8af7e4be907a584c4c5586219571c2 WatchSource:0}: Error finding container b6721b2b27f4f6916e50dacfb7c3c48c6f8af7e4be907a584c4c5586219571c2: Status 404 returned error can't find the container with id b6721b2b27f4f6916e50dacfb7c3c48c6f8af7e4be907a584c4c5586219571c2
Sep 11 11:49:04.277163 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:04.277133    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:49:04.277564 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:49:04.277313    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 11:49:04.277564 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:49:04.277393    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls podName:89c76d6c-b4b3-442a-8081-0ec81e5894a6 nodeName:}" failed. No retries permitted until 2026-09-11 11:49:36.277372301 +0000 UTC m=+98.699240680 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls") pod "dns-default-hmj9n" (UID: "89c76d6c-b4b3-442a-8081-0ec81e5894a6") : secret "dns-default-metrics-tls" not found
Sep 11 11:49:04.315346 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:04.315311    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p72gz" event={"ID":"83d5cf45-f64b-4e9d-8d77-088b52928c66","Type":"ContainerStarted","Data":"b6721b2b27f4f6916e50dacfb7c3c48c6f8af7e4be907a584c4c5586219571c2"}
Sep 11 11:49:04.316737 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:04.316709    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sm98f" event={"ID":"584f8d60-a99b-4ab3-acbe-c5de6343284f","Type":"ContainerStarted","Data":"cbee99c7775e042f268e515d3cffe18cd4c2a625a4af85467ec789cdb4594372"}
Sep 11 11:49:08.327708 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:08.327672    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p72gz" event={"ID":"83d5cf45-f64b-4e9d-8d77-088b52928c66","Type":"ContainerStarted","Data":"cbfcca3715c6dc3d1d43417ea451e8bcfb6318d0668ede2be3587c7292f0d5ff"}
Sep 11 11:49:08.328115 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:08.327798    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:49:08.329300 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:08.329275    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sm98f" event={"ID":"584f8d60-a99b-4ab3-acbe-c5de6343284f","Type":"ContainerStarted","Data":"15a1bc052cbad5be7029798f747e9508c8066b30b94d891cd7fcfe64685cfbd5"}
Sep 11 11:49:08.344216 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:08.344171    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-p72gz" podStartSLOduration=65.65019613 podStartE2EDuration="1m10.344154493s" totalImagesPullingTime="4.693958363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:49:08.344153671 +0000 UTC m=+70.766022095" watchObservedRunningTime="2026-09-11 11:49:08.344154493 +0000 UTC m=+70.766022853"
Sep 11 11:49:08.359291 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:08.359254    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sm98f" podStartSLOduration=65.56461254 podStartE2EDuration="1m10.359234715s" totalImagesPullingTime="4.794622175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:49:08.3586494 +0000 UTC m=+70.780517800" watchObservedRunningTime="2026-09-11 11:49:08.359234715 +0000 UTC m=+70.781103094"
Sep 11 11:49:36.373268 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:36.373187    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:49:36.373679 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:49:36.373291    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 11:49:36.373679 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:49:36.373341    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls podName:89c76d6c-b4b3-442a-8081-0ec81e5894a6 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:40.373328214 +0000 UTC m=+162.795196571 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls") pod "dns-default-hmj9n" (UID: "89c76d6c-b4b3-442a-8081-0ec81e5894a6") : secret "dns-default-metrics-tls" not found
Sep 11 11:49:39.334749 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:49:39.334709    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-p72gz"
Sep 11 11:50:04.510244 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.510209    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"]
Sep 11 11:50:04.512003 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.511985    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"
Sep 11 11:50:04.518647 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.518625    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 11 11:50:04.518647 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.518638    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 11:50:04.518895 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.518632    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 11:50:04.518895 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.518732    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mggcx\""
Sep 11 11:50:04.518895 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.518803    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 11 11:50:04.523663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.523640    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"]
Sep 11 11:50:04.544756 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.544730    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c1a888c2-3145-46ea-90b4-d026de4a6ad0-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-vdr7c\" (UID: \"c1a888c2-3145-46ea-90b4-d026de4a6ad0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"
Sep 11 11:50:04.544846 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.544763    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vdr7c\" (UID: \"c1a888c2-3145-46ea-90b4-d026de4a6ad0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"
Sep 11 11:50:04.544846 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.544789    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b6lm\" (UniqueName: \"kubernetes.io/projected/c1a888c2-3145-46ea-90b4-d026de4a6ad0-kube-api-access-2b6lm\") pod \"cluster-monitoring-operator-6cd954b897-vdr7c\" (UID: \"c1a888c2-3145-46ea-90b4-d026de4a6ad0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"
Sep 11 11:50:04.616561 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.616539    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn"]
Sep 11 11:50:04.618351 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.618336    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn"
Sep 11 11:50:04.621614 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.621597    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 11 11:50:04.621614 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.621609    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 11 11:50:04.621738 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.621717    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nbs7d\""
Sep 11 11:50:04.622138 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.622123    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 11 11:50:04.628385 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.628363    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd"]
Sep 11 11:50:04.629993 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.629973    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd"
Sep 11 11:50:04.633929 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.633910    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn"]
Sep 11 11:50:04.635029 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.635012    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 11 11:50:04.635321 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.635289    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 11 11:50:04.635437 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.635322    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 11 11:50:04.635437 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.635396    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qjpl6\""
Sep 11 11:50:04.637100 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.637080    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 11 11:50:04.645977 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.645957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b6lm\" (UniqueName: \"kubernetes.io/projected/c1a888c2-3145-46ea-90b4-d026de4a6ad0-kube-api-access-2b6lm\") pod \"cluster-monitoring-operator-6cd954b897-vdr7c\" (UID: \"c1a888c2-3145-46ea-90b4-d026de4a6ad0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"
Sep 11 11:50:04.646085 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.646008    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c1a888c2-3145-46ea-90b4-d026de4a6ad0-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-vdr7c\" (UID: \"c1a888c2-3145-46ea-90b4-d026de4a6ad0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"
Sep 11 11:50:04.646085 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.646042    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vdr7c\" (UID: \"c1a888c2-3145-46ea-90b4-d026de4a6ad0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"
Sep 11 11:50:04.646185 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:04.646124    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 11:50:04.646185 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:04.646181    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls podName:c1a888c2-3145-46ea-90b4-d026de4a6ad0 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:05.14616479 +0000 UTC m=+127.568033154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vdr7c" (UID: "c1a888c2-3145-46ea-90b4-d026de4a6ad0") : secret "cluster-monitoring-operator-tls" not found
Sep 11 11:50:04.647120 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.647061    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c1a888c2-3145-46ea-90b4-d026de4a6ad0-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-vdr7c\" (UID: \"c1a888c2-3145-46ea-90b4-d026de4a6ad0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"
Sep 11 11:50:04.647327 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.647297    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd"]
Sep 11 11:50:04.664259 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.664240    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b6lm\" (UniqueName: \"kubernetes.io/projected/c1a888c2-3145-46ea-90b4-d026de4a6ad0-kube-api-access-2b6lm\") pod \"cluster-monitoring-operator-6cd954b897-vdr7c\" (UID: \"c1a888c2-3145-46ea-90b4-d026de4a6ad0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"
Sep 11 11:50:04.724912 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.724890    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7ps58"]
Sep 11 11:50:04.726806 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.726791    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7ps58"
Sep 11 11:50:04.729046 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.729012    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-x7kgl\""
Sep 11 11:50:04.729264 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.729248    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 11 11:50:04.729472 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.729457    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 11 11:50:04.738292 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.738272    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7ps58"]
Sep 11 11:50:04.746675 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.746655    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nm9v\" (UniqueName: \"kubernetes.io/projected/236f4ed7-e91c-49de-8c80-8fb3490942c4-kube-api-access-7nm9v\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wk5vd\" (UID: \"236f4ed7-e91c-49de-8c80-8fb3490942c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd"
Sep 11 11:50:04.746757 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.746692    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/236f4ed7-e91c-49de-8c80-8fb3490942c4-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wk5vd\" (UID: \"236f4ed7-e91c-49de-8c80-8fb3490942c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd"
Sep 11 11:50:04.746796 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.746753    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-svhkn\" (UID: \"3406a424-94be-4995-be47-a8c4289f19dd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn"
Sep 11 11:50:04.746796 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.746779    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lrjs\" (UniqueName: \"kubernetes.io/projected/3406a424-94be-4995-be47-a8c4289f19dd-kube-api-access-9lrjs\") pod \"cluster-samples-operator-86499c89c4-svhkn\" (UID: \"3406a424-94be-4995-be47-a8c4289f19dd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn"
Sep 11 11:50:04.746877 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.746798    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/236f4ed7-e91c-49de-8c80-8fb3490942c4-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wk5vd\" (UID: \"236f4ed7-e91c-49de-8c80-8fb3490942c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd"
Sep 11 11:50:04.847928 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.847875    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-svhkn\" (UID: \"3406a424-94be-4995-be47-a8c4289f19dd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn"
Sep 11 11:50:04.847928 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.847901    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lrjs\" (UniqueName: \"kubernetes.io/projected/3406a424-94be-4995-be47-a8c4289f19dd-kube-api-access-9lrjs\") pod \"cluster-samples-operator-86499c89c4-svhkn\" (UID: \"3406a424-94be-4995-be47-a8c4289f19dd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn"
Sep 11 11:50:04.847928 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.847917    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/236f4ed7-e91c-49de-8c80-8fb3490942c4-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wk5vd\" (UID: \"236f4ed7-e91c-49de-8c80-8fb3490942c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd"
Sep 11 11:50:04.848119 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.847951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nm9v\" (UniqueName: \"kubernetes.io/projected/236f4ed7-e91c-49de-8c80-8fb3490942c4-kube-api-access-7nm9v\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wk5vd\" (UID: \"236f4ed7-e91c-49de-8c80-8fb3490942c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd"
Sep 11 11:50:04.848119 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.847975    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/236f4ed7-e91c-49de-8c80-8fb3490942c4-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wk5vd\" (UID: \"236f4ed7-e91c-49de-8c80-8fb3490942c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd"
Sep 11 11:50:04.848119 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.847991    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72qqb\" (UniqueName: \"kubernetes.io/projected/5b48c153-e731-4f3f-a4e6-254c18c4af63-kube-api-access-72qqb\") pod \"volume-data-source-validator-86695b7cc7-7ps58\" (UID: \"5b48c153-e731-4f3f-a4e6-254c18c4af63\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7ps58"
Sep 11 11:50:04.848119 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:04.848007    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 11:50:04.848119 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:04.848058    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls podName:3406a424-94be-4995-be47-a8c4289f19dd nodeName:}" failed. No retries permitted until 2026-09-11 11:50:05.348040165 +0000 UTC m=+127.769908524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-svhkn" (UID: "3406a424-94be-4995-be47-a8c4289f19dd") : secret "samples-operator-tls" not found
Sep 11 11:50:04.848493 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.848473    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/236f4ed7-e91c-49de-8c80-8fb3490942c4-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wk5vd\" (UID: \"236f4ed7-e91c-49de-8c80-8fb3490942c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd"
Sep 11 11:50:04.850135 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.850119    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/236f4ed7-e91c-49de-8c80-8fb3490942c4-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wk5vd\" (UID: \"236f4ed7-e91c-49de-8c80-8fb3490942c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd"
Sep 11 11:50:04.858738 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.858713    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nm9v\" (UniqueName: \"kubernetes.io/projected/236f4ed7-e91c-49de-8c80-8fb3490942c4-kube-api-access-7nm9v\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wk5vd\" (UID: \"236f4ed7-e91c-49de-8c80-8fb3490942c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd"
Sep 11 11:50:04.859738 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.859718    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lrjs\" (UniqueName: \"kubernetes.io/projected/3406a424-94be-4995-be47-a8c4289f19dd-kube-api-access-9lrjs\") pod \"cluster-samples-operator-86499c89c4-svhkn\" (UID: \"3406a424-94be-4995-be47-a8c4289f19dd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn"
Sep 11 11:50:04.939055 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.939036    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd"
Sep 11 11:50:04.948747 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.948728    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qqb\" (UniqueName: \"kubernetes.io/projected/5b48c153-e731-4f3f-a4e6-254c18c4af63-kube-api-access-72qqb\") pod \"volume-data-source-validator-86695b7cc7-7ps58\" (UID: \"5b48c153-e731-4f3f-a4e6-254c18c4af63\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7ps58"
Sep 11 11:50:04.959086 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:04.959068    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72qqb\" (UniqueName: \"kubernetes.io/projected/5b48c153-e731-4f3f-a4e6-254c18c4af63-kube-api-access-72qqb\") pod \"volume-data-source-validator-86695b7cc7-7ps58\" (UID: \"5b48c153-e731-4f3f-a4e6-254c18c4af63\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7ps58"
Sep 11 11:50:05.034597 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:05.034555    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7ps58"
Sep 11 11:50:05.150725 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:05.150660    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vdr7c\" (UID: \"c1a888c2-3145-46ea-90b4-d026de4a6ad0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"
Sep 11 11:50:05.150833 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:05.150780    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 11:50:05.150883 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:05.150872    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls podName:c1a888c2-3145-46ea-90b4-d026de4a6ad0 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:06.15084745 +0000 UTC m=+128.572715809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vdr7c" (UID: "c1a888c2-3145-46ea-90b4-d026de4a6ad0") : secret "cluster-monitoring-operator-tls" not found
Sep 11 11:50:05.205347 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:05.205324    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd"]
Sep 11 11:50:05.207945 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:50:05.207923    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236f4ed7_e91c_49de_8c80_8fb3490942c4.slice/crio-2a25d9319e789152096af2a93039570292807646013cf0ff9b4dc306cb0608be WatchSource:0}: Error finding container 2a25d9319e789152096af2a93039570292807646013cf0ff9b4dc306cb0608be: Status 404 returned error can't find the container with id 2a25d9319e789152096af2a93039570292807646013cf0ff9b4dc306cb0608be
Sep 11 11:50:05.297440 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:05.297414    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7ps58"]
Sep 11 11:50:05.299664 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:50:05.299640    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b48c153_e731_4f3f_a4e6_254c18c4af63.slice/crio-419ca6e4a16cc64c74dea12ccccd03ddb2e047d4c49d443ae00b46977fa36151 WatchSource:0}: Error finding container 419ca6e4a16cc64c74dea12ccccd03ddb2e047d4c49d443ae00b46977fa36151: Status 404 returned error can't find the container with id 419ca6e4a16cc64c74dea12ccccd03ddb2e047d4c49d443ae00b46977fa36151
Sep 11 11:50:05.352072 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:05.352047    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-svhkn\" (UID: \"3406a424-94be-4995-be47-a8c4289f19dd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn"
Sep 11 11:50:05.352178 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:05.352166    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 11:50:05.352227 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:05.352221    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls podName:3406a424-94be-4995-be47-a8c4289f19dd nodeName:}" failed. No retries permitted until 2026-09-11 11:50:06.35220753 +0000 UTC m=+128.774075888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-svhkn" (UID: "3406a424-94be-4995-be47-a8c4289f19dd") : secret "samples-operator-tls" not found
Sep 11 11:50:05.467989 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:05.467924    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd" event={"ID":"236f4ed7-e91c-49de-8c80-8fb3490942c4","Type":"ContainerStarted","Data":"2a25d9319e789152096af2a93039570292807646013cf0ff9b4dc306cb0608be"}
Sep 11 11:50:05.469001 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:05.468979    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7ps58" event={"ID":"5b48c153-e731-4f3f-a4e6-254c18c4af63","Type":"ContainerStarted","Data":"419ca6e4a16cc64c74dea12ccccd03ddb2e047d4c49d443ae00b46977fa36151"}
Sep 11 11:50:06.158243 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:06.158177    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vdr7c\" (UID: \"c1a888c2-3145-46ea-90b4-d026de4a6ad0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"
Sep 11 11:50:06.158738 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:06.158345    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 11:50:06.158738 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:06.158421    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls podName:c1a888c2-3145-46ea-90b4-d026de4a6ad0 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:08.158402601 +0000 UTC m=+130.580270961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vdr7c" (UID: "c1a888c2-3145-46ea-90b4-d026de4a6ad0") : secret "cluster-monitoring-operator-tls" not found
Sep 11 11:50:06.359029 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:06.358998    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-svhkn\" (UID: \"3406a424-94be-4995-be47-a8c4289f19dd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn"
Sep 11 11:50:06.359204 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:06.359168    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 11:50:06.359272 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:06.359251    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls podName:3406a424-94be-4995-be47-a8c4289f19dd nodeName:}" failed. No retries permitted until 2026-09-11 11:50:08.359229699 +0000 UTC m=+130.781098060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-svhkn" (UID: "3406a424-94be-4995-be47-a8c4289f19dd") : secret "samples-operator-tls" not found
Sep 11 11:50:06.963786 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:06.963701    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs\") pod \"network-metrics-daemon-2xwtg\" (UID: \"ca51005b-bb90-4eac-8bc7-c64bb5463c3f\") " pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:50:06.963944 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:06.963888    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 11:50:06.964014 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:06.963977    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs podName:ca51005b-bb90-4eac-8bc7-c64bb5463c3f nodeName:}" failed. No retries permitted until 2026-09-11 11:52:08.96395284 +0000 UTC m=+251.385821202 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs") pod "network-metrics-daemon-2xwtg" (UID: "ca51005b-bb90-4eac-8bc7-c64bb5463c3f") : secret "metrics-daemon-secret" not found
Sep 11 11:50:07.198561 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.198533    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69b567967-lrbqw"]
Sep 11 11:50:07.200270 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.200252    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.202646 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.202627    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 11:50:07.202761 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.202743    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 11:50:07.203424 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.203406    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-47c8b\""
Sep 11 11:50:07.206294 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.206264    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 11:50:07.210839 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.210819    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 11:50:07.220051 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.220006    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69b567967-lrbqw"]
Sep 11 11:50:07.266920 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.266901    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-certificates\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.267011 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.266930    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-ca-trust-extracted\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.267011 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.266947    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-bound-sa-token\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.267011 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.267002    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-image-registry-private-configuration\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.267106 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.267057    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.267106 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.267082    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-installation-pull-secrets\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.267166 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.267142    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-trusted-ca\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.267198 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.267165    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx2fd\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-kube-api-access-kx2fd\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.368416 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.368333    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-image-registry-private-configuration\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.368507 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.368461    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.368507 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.368489    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-installation-pull-secrets\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.368630 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.368517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-trusted-ca\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.368630 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.368539    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx2fd\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-kube-api-access-kx2fd\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.368630 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.368573    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-certificates\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.368630 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:07.368618    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 11:50:07.368882 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:07.368643    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b567967-lrbqw: secret "image-registry-tls" not found
Sep 11 11:50:07.368882 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:07.368717    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls podName:c7448b21-ef18-4ab5-bea4-cebb5d20fb13 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:07.868683878 +0000 UTC m=+130.290552251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls") pod "image-registry-69b567967-lrbqw" (UID: "c7448b21-ef18-4ab5-bea4-cebb5d20fb13") : secret "image-registry-tls" not found
Sep 11 11:50:07.368882 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.368623    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-ca-trust-extracted\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.368882 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.368770    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-bound-sa-token\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.369086 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.369001    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-ca-trust-extracted\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.369625 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.369581    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-certificates\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.369779 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.369763    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-trusted-ca\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.370893 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.370872    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-installation-pull-secrets\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.370959 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.370880    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-image-registry-private-configuration\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.385040 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.385020    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx2fd\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-kube-api-access-kx2fd\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.389628 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.389606    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-bound-sa-token\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.477965 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.477908    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd" event={"ID":"236f4ed7-e91c-49de-8c80-8fb3490942c4","Type":"ContainerStarted","Data":"d3b65d447dad0ad61269ac2b09251818bd6426c776b222f147a5c307b919b5c1"}
Sep 11 11:50:07.479414 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.479395    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7ps58" event={"ID":"5b48c153-e731-4f3f-a4e6-254c18c4af63","Type":"ContainerStarted","Data":"4d8429a66e7378e3d3d49a33a2204d53ffc16f3b23c5b4396beb4389dd6064b4"}
Sep 11 11:50:07.503335 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.503290    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd" podStartSLOduration=1.667416375 podStartE2EDuration="3.503276593s" totalImagesPullingTime="1.835860218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:50:07.500495997 +0000 UTC m=+129.922364407" watchObservedRunningTime="2026-09-11 11:50:07.503276593 +0000 UTC m=+129.925145029"
Sep 11 11:50:07.534818 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.534774    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7ps58" podStartSLOduration=2.27723545 podStartE2EDuration="3.534757818s" totalImagesPullingTime="1.257522368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:50:07.532127953 +0000 UTC m=+129.953996333" watchObservedRunningTime="2026-09-11 11:50:07.534757818 +0000 UTC m=+129.956626200"
Sep 11 11:50:07.873334 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:07.873309    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:07.873450 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:07.873432    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 11:50:07.873450 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:07.873450    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b567967-lrbqw: secret "image-registry-tls" not found
Sep 11 11:50:07.873525 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:07.873496    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls podName:c7448b21-ef18-4ab5-bea4-cebb5d20fb13 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:08.873483181 +0000 UTC m=+131.295351538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls") pod "image-registry-69b567967-lrbqw" (UID: "c7448b21-ef18-4ab5-bea4-cebb5d20fb13") : secret "image-registry-tls" not found
Sep 11 11:50:08.175297 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:08.175218    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vdr7c\" (UID: \"c1a888c2-3145-46ea-90b4-d026de4a6ad0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"
Sep 11 11:50:08.175424 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:08.175326    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 11:50:08.175424 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:08.175383    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls podName:c1a888c2-3145-46ea-90b4-d026de4a6ad0 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:12.175369216 +0000 UTC m=+134.597237574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vdr7c" (UID: "c1a888c2-3145-46ea-90b4-d026de4a6ad0") : secret "cluster-monitoring-operator-tls" not found
Sep 11 11:50:08.376867 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:08.376833    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-svhkn\" (UID: \"3406a424-94be-4995-be47-a8c4289f19dd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn"
Sep 11 11:50:08.377225 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:08.376973    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 11:50:08.377225 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:08.377031    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls podName:3406a424-94be-4995-be47-a8c4289f19dd nodeName:}" failed. No retries permitted until 2026-09-11 11:50:12.377015037 +0000 UTC m=+134.798883415 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-svhkn" (UID: "3406a424-94be-4995-be47-a8c4289f19dd") : secret "samples-operator-tls" not found
Sep 11 11:50:08.880924 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:08.880892    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:08.881089 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:08.881023    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 11:50:08.881089 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:08.881042    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b567967-lrbqw: secret "image-registry-tls" not found
Sep 11 11:50:08.881189 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:08.881110    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls podName:c7448b21-ef18-4ab5-bea4-cebb5d20fb13 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:10.881091666 +0000 UTC m=+133.302960039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls") pod "image-registry-69b567967-lrbqw" (UID: "c7448b21-ef18-4ab5-bea4-cebb5d20fb13") : secret "image-registry-tls" not found
Sep 11 11:50:09.732675 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.732644    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-859g8"]
Sep 11 11:50:09.735167 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.735150    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:09.740613 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.740574    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5tl7g\""
Sep 11 11:50:09.740713 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.740674    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 11:50:09.742659 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.742640    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 11:50:09.742955 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.742939    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 11:50:09.743037 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.742987    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 11:50:09.750241 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.750221    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-859g8"]
Sep 11 11:50:09.786561 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.786535    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8592c40b-2ab2-47ee-872b-4aa604fe5b36-crio-socket\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:09.786676 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.786627    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8592c40b-2ab2-47ee-872b-4aa604fe5b36-data-volume\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:09.786676 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.786661    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:09.786797 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.786684    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rmcm\" (UniqueName: \"kubernetes.io/projected/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-api-access-7rmcm\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:09.786797 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.786753    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:09.887244 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.887217    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8592c40b-2ab2-47ee-872b-4aa604fe5b36-crio-socket\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:09.887362 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.887272    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8592c40b-2ab2-47ee-872b-4aa604fe5b36-data-volume\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:09.887362 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.887347    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8592c40b-2ab2-47ee-872b-4aa604fe5b36-crio-socket\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:09.887459 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.887389    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:09.887459 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.887425    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rmcm\" (UniqueName: \"kubernetes.io/projected/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-api-access-7rmcm\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:09.887459 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.887454    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:09.887603 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:09.887486    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 11:50:09.887603 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.887499    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8592c40b-2ab2-47ee-872b-4aa604fe5b36-data-volume\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:09.887603 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:09.887551    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls podName:8592c40b-2ab2-47ee-872b-4aa604fe5b36 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:10.387533196 +0000 UTC m=+132.809401555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-859g8" (UID: "8592c40b-2ab2-47ee-872b-4aa604fe5b36") : secret "insights-runtime-extractor-tls" not found
Sep 11 11:50:09.887793 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:09.887671    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 11:50:09.887793 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:09.887739    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm podName:8592c40b-2ab2-47ee-872b-4aa604fe5b36 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:10.387730275 +0000 UTC m=+132.809598659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-859g8" (UID: "8592c40b-2ab2-47ee-872b-4aa604fe5b36") : configmap "kube-rbac-proxy" not found
Sep 11 11:50:09.896464 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:09.896445    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rmcm\" (UniqueName: \"kubernetes.io/projected/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-api-access-7rmcm\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:10.392618 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:10.392570    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:10.392618 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:10.392621    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:10.392818 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:10.392682    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 11:50:10.392818 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:10.392710    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 11:50:10.392818 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:10.392773    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls podName:8592c40b-2ab2-47ee-872b-4aa604fe5b36 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:11.392758188 +0000 UTC m=+133.814626546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-859g8" (UID: "8592c40b-2ab2-47ee-872b-4aa604fe5b36") : secret "insights-runtime-extractor-tls" not found
Sep 11 11:50:10.392818 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:10.392788    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm podName:8592c40b-2ab2-47ee-872b-4aa604fe5b36 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:11.392781487 +0000 UTC m=+133.814649845 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-859g8" (UID: "8592c40b-2ab2-47ee-872b-4aa604fe5b36") : configmap "kube-rbac-proxy" not found
Sep 11 11:50:10.896852 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:10.896819    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:10.897234 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:10.896941    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 11:50:10.897234 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:10.896952    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b567967-lrbqw: secret "image-registry-tls" not found
Sep 11 11:50:10.897234 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:10.897004    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls podName:c7448b21-ef18-4ab5-bea4-cebb5d20fb13 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:14.896990978 +0000 UTC m=+137.318859348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls") pod "image-registry-69b567967-lrbqw" (UID: "c7448b21-ef18-4ab5-bea4-cebb5d20fb13") : secret "image-registry-tls" not found
Sep 11 11:50:11.401575 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:11.401541    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:11.401767 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:11.401655    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 11:50:11.401767 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:11.401684    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:11.401767 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:11.401710    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm podName:8592c40b-2ab2-47ee-872b-4aa604fe5b36 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:13.401697171 +0000 UTC m=+135.823565529 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-859g8" (UID: "8592c40b-2ab2-47ee-872b-4aa604fe5b36") : configmap "kube-rbac-proxy" not found
Sep 11 11:50:11.401922 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:11.401780    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 11:50:11.401922 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:11.401839    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls podName:8592c40b-2ab2-47ee-872b-4aa604fe5b36 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:13.401823486 +0000 UTC m=+135.823691860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-859g8" (UID: "8592c40b-2ab2-47ee-872b-4aa604fe5b36") : secret "insights-runtime-extractor-tls" not found
Sep 11 11:50:12.207936 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:12.207895    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vdr7c\" (UID: \"c1a888c2-3145-46ea-90b4-d026de4a6ad0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"
Sep 11 11:50:12.208452 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:12.208054    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 11:50:12.208452 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:12.208134    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls podName:c1a888c2-3145-46ea-90b4-d026de4a6ad0 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:20.208112778 +0000 UTC m=+142.629981144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vdr7c" (UID: "c1a888c2-3145-46ea-90b4-d026de4a6ad0") : secret "cluster-monitoring-operator-tls" not found
Sep 11 11:50:12.409060 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:12.409020    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-svhkn\" (UID: \"3406a424-94be-4995-be47-a8c4289f19dd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn"
Sep 11 11:50:12.409210 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:12.409172    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 11:50:12.409249 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:12.409237    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls podName:3406a424-94be-4995-be47-a8c4289f19dd nodeName:}" failed. No retries permitted until 2026-09-11 11:50:20.409222766 +0000 UTC m=+142.831091125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-svhkn" (UID: "3406a424-94be-4995-be47-a8c4289f19dd") : secret "samples-operator-tls" not found
Sep 11 11:50:12.663422 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:12.663405    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vk54k_714f5e6f-c22d-48de-b545-6be2e58968b0/dns-node-resolver/0.log"
Sep 11 11:50:12.980814 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:12.980743    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-fj5dp"]
Sep 11 11:50:12.983125 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:12.983110    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-fj5dp"
Sep 11 11:50:12.985711 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:12.985689    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 11:50:12.986339 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:12.986318    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 11:50:12.986466 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:12.986440    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 11:50:12.986466 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:12.986463    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 11:50:12.986617 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:12.986451    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jkwnd\""
Sep 11 11:50:12.992996 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:12.992975    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-fj5dp"]
Sep 11 11:50:13.113668 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:13.113643    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/820edd47-9fc1-4cc2-9960-d2d951636abd-signing-cabundle\") pod \"service-ca-568869fdc5-fj5dp\" (UID: \"820edd47-9fc1-4cc2-9960-d2d951636abd\") " pod="openshift-service-ca/service-ca-568869fdc5-fj5dp"
Sep 11 11:50:13.113780 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:13.113711    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzvft\" (UniqueName: \"kubernetes.io/projected/820edd47-9fc1-4cc2-9960-d2d951636abd-kube-api-access-tzvft\") pod \"service-ca-568869fdc5-fj5dp\" (UID: \"820edd47-9fc1-4cc2-9960-d2d951636abd\") " pod="openshift-service-ca/service-ca-568869fdc5-fj5dp"
Sep 11 11:50:13.113857 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:13.113814    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/820edd47-9fc1-4cc2-9960-d2d951636abd-signing-key\") pod \"service-ca-568869fdc5-fj5dp\" (UID: \"820edd47-9fc1-4cc2-9960-d2d951636abd\") " pod="openshift-service-ca/service-ca-568869fdc5-fj5dp"
Sep 11 11:50:13.214829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:13.214804    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/820edd47-9fc1-4cc2-9960-d2d951636abd-signing-key\") pod \"service-ca-568869fdc5-fj5dp\" (UID: \"820edd47-9fc1-4cc2-9960-d2d951636abd\") " pod="openshift-service-ca/service-ca-568869fdc5-fj5dp"
Sep 11 11:50:13.215189 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:13.214868    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/820edd47-9fc1-4cc2-9960-d2d951636abd-signing-cabundle\") pod \"service-ca-568869fdc5-fj5dp\" (UID: \"820edd47-9fc1-4cc2-9960-d2d951636abd\") " pod="openshift-service-ca/service-ca-568869fdc5-fj5dp"
Sep 11 11:50:13.215189 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:13.214996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzvft\" (UniqueName: \"kubernetes.io/projected/820edd47-9fc1-4cc2-9960-d2d951636abd-kube-api-access-tzvft\") pod \"service-ca-568869fdc5-fj5dp\" (UID: \"820edd47-9fc1-4cc2-9960-d2d951636abd\") " pod="openshift-service-ca/service-ca-568869fdc5-fj5dp"
Sep 11 11:50:13.215451 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:13.215431    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/820edd47-9fc1-4cc2-9960-d2d951636abd-signing-cabundle\") pod \"service-ca-568869fdc5-fj5dp\" (UID: \"820edd47-9fc1-4cc2-9960-d2d951636abd\") " pod="openshift-service-ca/service-ca-568869fdc5-fj5dp"
Sep 11 11:50:13.217227 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:13.217209    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/820edd47-9fc1-4cc2-9960-d2d951636abd-signing-key\") pod \"service-ca-568869fdc5-fj5dp\" (UID: \"820edd47-9fc1-4cc2-9960-d2d951636abd\") " pod="openshift-service-ca/service-ca-568869fdc5-fj5dp"
Sep 11 11:50:13.224461 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:13.224444    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzvft\" (UniqueName: \"kubernetes.io/projected/820edd47-9fc1-4cc2-9960-d2d951636abd-kube-api-access-tzvft\") pod \"service-ca-568869fdc5-fj5dp\" (UID: \"820edd47-9fc1-4cc2-9960-d2d951636abd\") " pod="openshift-service-ca/service-ca-568869fdc5-fj5dp"
Sep 11 11:50:13.291361 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:13.291343    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-fj5dp"
Sep 11 11:50:13.416636 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:13.416603    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:13.416757 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:13.416653    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:13.416757 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:13.416746    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 11:50:13.416847 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:13.416753    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 11:50:13.416847 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:13.416814    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls podName:8592c40b-2ab2-47ee-872b-4aa604fe5b36 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:17.416797969 +0000 UTC m=+139.838666328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-859g8" (UID: "8592c40b-2ab2-47ee-872b-4aa604fe5b36") : secret "insights-runtime-extractor-tls" not found
Sep 11 11:50:13.416847 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:13.416831    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm podName:8592c40b-2ab2-47ee-872b-4aa604fe5b36 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:17.416823546 +0000 UTC m=+139.838691904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-859g8" (UID: "8592c40b-2ab2-47ee-872b-4aa604fe5b36") : configmap "kube-rbac-proxy" not found
Sep 11 11:50:13.468098 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:13.468077    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fjbdx_96f46a0c-d972-4490-a888-ca0f79174b50/node-ca/0.log"
Sep 11 11:50:13.558846 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:13.558781    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-fj5dp"]
Sep 11 11:50:13.562999 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:50:13.562974    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820edd47_9fc1_4cc2_9960_d2d951636abd.slice/crio-39c81494f8e6a77e0d058e57428bac9224f24d7b4da53a1593995c38fecf8dc5 WatchSource:0}: Error finding container 39c81494f8e6a77e0d058e57428bac9224f24d7b4da53a1593995c38fecf8dc5: Status 404 returned error can't find the container with id 39c81494f8e6a77e0d058e57428bac9224f24d7b4da53a1593995c38fecf8dc5
Sep 11 11:50:14.279415 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:14.279385    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c4qfd_775678cb-ae06-458a-a9f1-1ab4b5f48928/serve-healthcheck-canary/0.log"
Sep 11 11:50:14.502430 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:14.502393    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-fj5dp" event={"ID":"820edd47-9fc1-4cc2-9960-d2d951636abd","Type":"ContainerStarted","Data":"39c81494f8e6a77e0d058e57428bac9224f24d7b4da53a1593995c38fecf8dc5"}
Sep 11 11:50:14.927699 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:14.927661    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:14.927877 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:14.927837    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 11:50:14.927877 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:14.927859    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b567967-lrbqw: secret "image-registry-tls" not found
Sep 11 11:50:14.927974 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:14.927941    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls podName:c7448b21-ef18-4ab5-bea4-cebb5d20fb13 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:22.927919031 +0000 UTC m=+145.349787403 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls") pod "image-registry-69b567967-lrbqw" (UID: "c7448b21-ef18-4ab5-bea4-cebb5d20fb13") : secret "image-registry-tls" not found
Sep 11 11:50:15.067271 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:15.067234    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-wk5vd_236f4ed7-e91c-49de-8c80-8fb3490942c4/kube-storage-version-migrator-operator/0.log"
Sep 11 11:50:15.506818 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:15.506786    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-fj5dp" event={"ID":"820edd47-9fc1-4cc2-9960-d2d951636abd","Type":"ContainerStarted","Data":"bd8eaa51973ee86987789282c83742eed658d772a928697e86b56b354cb4bd6c"}
Sep 11 11:50:15.527153 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:15.527100    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-fj5dp" podStartSLOduration=1.900656014 podStartE2EDuration="3.527058702s" totalImagesPullingTime="1.626402688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:50:15.526309875 +0000 UTC m=+137.948178274" watchObservedRunningTime="2026-09-11 11:50:15.527058702 +0000 UTC m=+137.948927082"
Sep 11 11:50:17.446195 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:17.446093    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:17.446195 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:17.446187    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 11:50:17.446629 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:17.446250    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:17.446629 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:17.446324    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 11:50:17.446629 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:17.446349    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls podName:8592c40b-2ab2-47ee-872b-4aa604fe5b36 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:25.446323661 +0000 UTC m=+147.868192034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-859g8" (UID: "8592c40b-2ab2-47ee-872b-4aa604fe5b36") : secret "insights-runtime-extractor-tls" not found
Sep 11 11:50:17.446629 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:17.446382    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm podName:8592c40b-2ab2-47ee-872b-4aa604fe5b36 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:25.446364903 +0000 UTC m=+147.868233267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-859g8" (UID: "8592c40b-2ab2-47ee-872b-4aa604fe5b36") : configmap "kube-rbac-proxy" not found
Sep 11 11:50:20.266272 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:20.266242    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vdr7c\" (UID: \"c1a888c2-3145-46ea-90b4-d026de4a6ad0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"
Sep 11 11:50:20.266654 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:20.266395    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 11:50:20.266654 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:20.266466    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls podName:c1a888c2-3145-46ea-90b4-d026de4a6ad0 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:36.266447254 +0000 UTC m=+158.688315617 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vdr7c" (UID: "c1a888c2-3145-46ea-90b4-d026de4a6ad0") : secret "cluster-monitoring-operator-tls" not found
Sep 11 11:50:20.468033 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:20.467999    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-svhkn\" (UID: \"3406a424-94be-4995-be47-a8c4289f19dd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn"
Sep 11 11:50:20.470472 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:20.470441    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3406a424-94be-4995-be47-a8c4289f19dd-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-svhkn\" (UID: \"3406a424-94be-4995-be47-a8c4289f19dd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn"
Sep 11 11:50:20.526381 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:20.526330    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn"
Sep 11 11:50:20.801215 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:20.801153    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn"]
Sep 11 11:50:21.524250 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:21.524216    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn" event={"ID":"3406a424-94be-4995-be47-a8c4289f19dd","Type":"ContainerStarted","Data":"5cde0ae2cc3eb8b009ae56b40b38f265495bd85eee46e8bb16e4656b019af4ac"}
Sep 11 11:50:22.986979 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:22.986948    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:22.989055 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:22.989025    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls\") pod \"image-registry-69b567967-lrbqw\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") " pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:23.109361 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:23.109336    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:23.385727 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:23.385658    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69b567967-lrbqw"]
Sep 11 11:50:23.389022 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:50:23.388997    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7448b21_ef18_4ab5_bea4_cebb5d20fb13.slice/crio-5cbf0e730e4d75ea354b40cb16f0a650700d9b2ae4aa5263f1ee608075102a2a WatchSource:0}: Error finding container 5cbf0e730e4d75ea354b40cb16f0a650700d9b2ae4aa5263f1ee608075102a2a: Status 404 returned error can't find the container with id 5cbf0e730e4d75ea354b40cb16f0a650700d9b2ae4aa5263f1ee608075102a2a
Sep 11 11:50:23.534313 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:23.534278    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn" event={"ID":"3406a424-94be-4995-be47-a8c4289f19dd","Type":"ContainerStarted","Data":"0b4a13d90fd3a63dab3d0e6bc61bf9d1724bd28bd2a1795e5bab6d30262858fa"}
Sep 11 11:50:23.534492 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:23.534336    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn" event={"ID":"3406a424-94be-4995-be47-a8c4289f19dd","Type":"ContainerStarted","Data":"9d9dcd2c9af095ca981cea69420160797b53d7f276f11feb31f68e6e1b958d3b"}
Sep 11 11:50:23.535879 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:23.535854    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69b567967-lrbqw" event={"ID":"c7448b21-ef18-4ab5-bea4-cebb5d20fb13","Type":"ContainerStarted","Data":"ed43f7255310d269c8141bc7ca38014019b4dab1390ef5930fbbd0375e442c0c"}
Sep 11 11:50:23.535988 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:23.535895    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69b567967-lrbqw" event={"ID":"c7448b21-ef18-4ab5-bea4-cebb5d20fb13","Type":"ContainerStarted","Data":"5cbf0e730e4d75ea354b40cb16f0a650700d9b2ae4aa5263f1ee608075102a2a"}
Sep 11 11:50:23.535988 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:23.535973    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:23.554868 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:23.554819    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-svhkn" podStartSLOduration=17.536041444 podStartE2EDuration="19.554804816s" totalImagesPullingTime="2.018763372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:50:23.553988035 +0000 UTC m=+145.975856440" watchObservedRunningTime="2026-09-11 11:50:23.554804816 +0000 UTC m=+145.976673195"
Sep 11 11:50:23.575693 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:23.575646    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69b567967-lrbqw" podStartSLOduration=16.575632864 podStartE2EDuration="16.575632864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:50:23.574495168 +0000 UTC m=+145.996363568" watchObservedRunningTime="2026-09-11 11:50:23.575632864 +0000 UTC m=+145.997501243"
Sep 11 11:50:25.505077 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:25.505034    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:25.505449 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:25.505097    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:25.505449 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:25.505189    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 11:50:25.505449 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:25.505259    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls podName:8592c40b-2ab2-47ee-872b-4aa604fe5b36 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:41.505240589 +0000 UTC m=+163.927108965 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-859g8" (UID: "8592c40b-2ab2-47ee-872b-4aa604fe5b36") : secret "insights-runtime-extractor-tls" not found
Sep 11 11:50:25.505688 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:25.505669    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8592c40b-2ab2-47ee-872b-4aa604fe5b36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:35.502676 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:35.502625    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hmj9n" podUID="89c76d6c-b4b3-442a-8081-0ec81e5894a6"
Sep 11 11:50:35.572032 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:35.572005    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:50:35.907890 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:35.907820    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69b567967-lrbqw"]
Sep 11 11:50:35.975857 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:35.975830    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c869f9486-nt2v4"]
Sep 11 11:50:35.980771 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:35.980755    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:35.993517 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:35.993498    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c869f9486-nt2v4"]
Sep 11 11:50:36.082567 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.082545    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg252\" (UniqueName: \"kubernetes.io/projected/da3036fd-d0f4-44b4-876e-be8d5971886e-kube-api-access-xg252\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.082668 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.082577    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da3036fd-d0f4-44b4-876e-be8d5971886e-bound-sa-token\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.082668 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.082616    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da3036fd-d0f4-44b4-876e-be8d5971886e-installation-pull-secrets\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.082668 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.082644    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da3036fd-d0f4-44b4-876e-be8d5971886e-trusted-ca\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.082784 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.082670    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da3036fd-d0f4-44b4-876e-be8d5971886e-image-registry-private-configuration\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.082784 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.082686    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3036fd-d0f4-44b4-876e-be8d5971886e-registry-tls\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.082784 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.082724    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da3036fd-d0f4-44b4-876e-be8d5971886e-ca-trust-extracted\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.082784 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.082758    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da3036fd-d0f4-44b4-876e-be8d5971886e-registry-certificates\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.183380 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.183328    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3036fd-d0f4-44b4-876e-be8d5971886e-registry-tls\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.183380 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.183360    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da3036fd-d0f4-44b4-876e-be8d5971886e-ca-trust-extracted\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.183561 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.183508    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da3036fd-d0f4-44b4-876e-be8d5971886e-registry-certificates\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.183648 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.183569    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg252\" (UniqueName: \"kubernetes.io/projected/da3036fd-d0f4-44b4-876e-be8d5971886e-kube-api-access-xg252\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.183648 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.183640    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da3036fd-d0f4-44b4-876e-be8d5971886e-bound-sa-token\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.183752 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.183675    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da3036fd-d0f4-44b4-876e-be8d5971886e-installation-pull-secrets\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.183752 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.183719    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da3036fd-d0f4-44b4-876e-be8d5971886e-trusted-ca\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.183858 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.183756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da3036fd-d0f4-44b4-876e-be8d5971886e-image-registry-private-configuration\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.184012 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.183975    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da3036fd-d0f4-44b4-876e-be8d5971886e-ca-trust-extracted\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.184319 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.184299    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da3036fd-d0f4-44b4-876e-be8d5971886e-registry-certificates\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.184984 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.184959    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da3036fd-d0f4-44b4-876e-be8d5971886e-trusted-ca\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.185961 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.185935    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3036fd-d0f4-44b4-876e-be8d5971886e-registry-tls\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.186048 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.186009    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da3036fd-d0f4-44b4-876e-be8d5971886e-installation-pull-secrets\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.186288 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.186270    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da3036fd-d0f4-44b4-876e-be8d5971886e-image-registry-private-configuration\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.196358 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.196337    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da3036fd-d0f4-44b4-876e-be8d5971886e-bound-sa-token\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.196480 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.196461    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg252\" (UniqueName: \"kubernetes.io/projected/da3036fd-d0f4-44b4-876e-be8d5971886e-kube-api-access-xg252\") pod \"image-registry-7c869f9486-nt2v4\" (UID: \"da3036fd-d0f4-44b4-876e-be8d5971886e\") " pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.284962 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.284936    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vdr7c\" (UID: \"c1a888c2-3145-46ea-90b4-d026de4a6ad0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"
Sep 11 11:50:36.286843 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.286824    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c1a888c2-3145-46ea-90b4-d026de4a6ad0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vdr7c\" (UID: \"c1a888c2-3145-46ea-90b4-d026de4a6ad0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"
Sep 11 11:50:36.289755 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.289733    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:36.321002 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.320979    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"
Sep 11 11:50:36.562433 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.562410    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c869f9486-nt2v4"]
Sep 11 11:50:36.564654 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:50:36.564628    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3036fd_d0f4_44b4_876e_be8d5971886e.slice/crio-8fd85b6e6fa30c0538bd689f62e2632bd316ed702aa021ebcf0b3ec0cb8c7b85 WatchSource:0}: Error finding container 8fd85b6e6fa30c0538bd689f62e2632bd316ed702aa021ebcf0b3ec0cb8c7b85: Status 404 returned error can't find the container with id 8fd85b6e6fa30c0538bd689f62e2632bd316ed702aa021ebcf0b3ec0cb8c7b85
Sep 11 11:50:36.576432 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.576408    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c869f9486-nt2v4" event={"ID":"da3036fd-d0f4-44b4-876e-be8d5971886e","Type":"ContainerStarted","Data":"8fd85b6e6fa30c0538bd689f62e2632bd316ed702aa021ebcf0b3ec0cb8c7b85"}
Sep 11 11:50:36.596290 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:36.596224    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c"]
Sep 11 11:50:36.598004 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:50:36.597982    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a888c2_3145_46ea_90b4_d026de4a6ad0.slice/crio-332a9417210c19cad10ae55e609780920f7ac9be4b8376c0fec650d551c2ca1a WatchSource:0}: Error finding container 332a9417210c19cad10ae55e609780920f7ac9be4b8376c0fec650d551c2ca1a: Status 404 returned error can't find the container with id 332a9417210c19cad10ae55e609780920f7ac9be4b8376c0fec650d551c2ca1a
Sep 11 11:50:37.079695 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:37.079658    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2xwtg" podUID="ca51005b-bb90-4eac-8bc7-c64bb5463c3f"
Sep 11 11:50:37.584308 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:37.584267    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c" event={"ID":"c1a888c2-3145-46ea-90b4-d026de4a6ad0","Type":"ContainerStarted","Data":"332a9417210c19cad10ae55e609780920f7ac9be4b8376c0fec650d551c2ca1a"}
Sep 11 11:50:37.586295 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:37.586264    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c869f9486-nt2v4" event={"ID":"da3036fd-d0f4-44b4-876e-be8d5971886e","Type":"ContainerStarted","Data":"214ba1674c7401559de4547f9436fdacaeecd6a03a91b04a851cf73620be609e"}
Sep 11 11:50:37.586443 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:37.586385    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:50:39.594223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:39.594185    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c" event={"ID":"c1a888c2-3145-46ea-90b4-d026de4a6ad0","Type":"ContainerStarted","Data":"5061e80e157728419be2c77af40933b2d0017493c8d09c3fbaa07aba12b16c3d"}
Sep 11 11:50:39.614998 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:39.614951    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c869f9486-nt2v4" podStartSLOduration=4.614936961 podStartE2EDuration="4.614936961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:50:37.609799839 +0000 UTC m=+160.031668239" watchObservedRunningTime="2026-09-11 11:50:39.614936961 +0000 UTC m=+162.036805340"
Sep 11 11:50:39.615286 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:39.615243    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vdr7c" podStartSLOduration=33.650584528 podStartE2EDuration="35.615230163s" totalImagesPullingTime="1.964645635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:50:39.614229935 +0000 UTC m=+162.036098334" watchObservedRunningTime="2026-09-11 11:50:39.615230163 +0000 UTC m=+162.037098545"
Sep 11 11:50:40.417120 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:40.417087    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:50:40.419982 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:40.419954    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89c76d6c-b4b3-442a-8081-0ec81e5894a6-metrics-tls\") pod \"dns-default-hmj9n\" (UID: \"89c76d6c-b4b3-442a-8081-0ec81e5894a6\") " pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:50:40.675263 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:40.675199    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m9sw5\""
Sep 11 11:50:40.683471 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:40.683445    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:50:40.959570 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:40.959503    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hmj9n"]
Sep 11 11:50:40.962707 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:50:40.962681    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c76d6c_b4b3_442a_8081_0ec81e5894a6.slice/crio-cc6b1dec18166c97cea4ecebb8f1b523e612b7d075ccf5d03559f76fd657779a WatchSource:0}: Error finding container cc6b1dec18166c97cea4ecebb8f1b523e612b7d075ccf5d03559f76fd657779a: Status 404 returned error can't find the container with id cc6b1dec18166c97cea4ecebb8f1b523e612b7d075ccf5d03559f76fd657779a
Sep 11 11:50:41.524424 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:41.524380    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:41.526718 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:41.526695    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8592c40b-2ab2-47ee-872b-4aa604fe5b36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-859g8\" (UID: \"8592c40b-2ab2-47ee-872b-4aa604fe5b36\") " pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:41.544692 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:41.544675    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-859g8"
Sep 11 11:50:41.603748 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:41.603713    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmj9n" event={"ID":"89c76d6c-b4b3-442a-8081-0ec81e5894a6","Type":"ContainerStarted","Data":"cc6b1dec18166c97cea4ecebb8f1b523e612b7d075ccf5d03559f76fd657779a"}
Sep 11 11:50:41.820964 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:41.820893    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-859g8"]
Sep 11 11:50:41.823925 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:50:41.823886    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8592c40b_2ab2_47ee_872b_4aa604fe5b36.slice/crio-1bf1618f9b73c457a708ea4afcf217fdfb381dc1f89ac66d4420934f9f0713c2 WatchSource:0}: Error finding container 1bf1618f9b73c457a708ea4afcf217fdfb381dc1f89ac66d4420934f9f0713c2: Status 404 returned error can't find the container with id 1bf1618f9b73c457a708ea4afcf217fdfb381dc1f89ac66d4420934f9f0713c2
Sep 11 11:50:42.609161 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:42.609004    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-859g8" event={"ID":"8592c40b-2ab2-47ee-872b-4aa604fe5b36","Type":"ContainerStarted","Data":"eeb937d0558642e3fbbb00bc2bf5e047baef9d5772e29c9149939724ac3da53f"}
Sep 11 11:50:42.609161 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:42.609062    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-859g8" event={"ID":"8592c40b-2ab2-47ee-872b-4aa604fe5b36","Type":"ContainerStarted","Data":"1bf1618f9b73c457a708ea4afcf217fdfb381dc1f89ac66d4420934f9f0713c2"}
Sep 11 11:50:43.216712 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.216643    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-2snb2"]
Sep 11 11:50:43.219646 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.219630    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2"
Sep 11 11:50:43.223109 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.223086    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rws95\""
Sep 11 11:50:43.223230 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.223109    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 11:50:43.223230 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.223125    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 11:50:43.223323 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.223231    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 11:50:43.229407 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.229372    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-2snb2"]
Sep 11 11:50:43.340029 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.340003    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4h8f\" (UniqueName: \"kubernetes.io/projected/b3231f7c-5ad1-4376-9bb2-3dec096cb340-kube-api-access-p4h8f\") pod \"prometheus-operator-68bcd7f987-2snb2\" (UID: \"b3231f7c-5ad1-4376-9bb2-3dec096cb340\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2"
Sep 11 11:50:43.340150 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.340050    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3231f7c-5ad1-4376-9bb2-3dec096cb340-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-2snb2\" (UID: \"b3231f7c-5ad1-4376-9bb2-3dec096cb340\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2"
Sep 11 11:50:43.340150 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.340077    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3231f7c-5ad1-4376-9bb2-3dec096cb340-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-2snb2\" (UID: \"b3231f7c-5ad1-4376-9bb2-3dec096cb340\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2"
Sep 11 11:50:43.340150 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.340103    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3231f7c-5ad1-4376-9bb2-3dec096cb340-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-2snb2\" (UID: \"b3231f7c-5ad1-4376-9bb2-3dec096cb340\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2"
Sep 11 11:50:43.440993 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.440963    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3231f7c-5ad1-4376-9bb2-3dec096cb340-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-2snb2\" (UID: \"b3231f7c-5ad1-4376-9bb2-3dec096cb340\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2"
Sep 11 11:50:43.441099 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.440999    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3231f7c-5ad1-4376-9bb2-3dec096cb340-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-2snb2\" (UID: \"b3231f7c-5ad1-4376-9bb2-3dec096cb340\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2"
Sep 11 11:50:43.441099 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.441064    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4h8f\" (UniqueName: \"kubernetes.io/projected/b3231f7c-5ad1-4376-9bb2-3dec096cb340-kube-api-access-p4h8f\") pod \"prometheus-operator-68bcd7f987-2snb2\" (UID: \"b3231f7c-5ad1-4376-9bb2-3dec096cb340\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2"
Sep 11 11:50:43.441099 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.441090    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3231f7c-5ad1-4376-9bb2-3dec096cb340-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-2snb2\" (UID: \"b3231f7c-5ad1-4376-9bb2-3dec096cb340\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2"
Sep 11 11:50:43.441648 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.441627    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3231f7c-5ad1-4376-9bb2-3dec096cb340-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-2snb2\" (UID: \"b3231f7c-5ad1-4376-9bb2-3dec096cb340\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2"
Sep 11 11:50:43.443476 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.443456    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3231f7c-5ad1-4376-9bb2-3dec096cb340-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-2snb2\" (UID: \"b3231f7c-5ad1-4376-9bb2-3dec096cb340\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2"
Sep 11 11:50:43.443642 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.443619    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3231f7c-5ad1-4376-9bb2-3dec096cb340-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-2snb2\" (UID: \"b3231f7c-5ad1-4376-9bb2-3dec096cb340\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2"
Sep 11 11:50:43.449993 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.449967    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4h8f\" (UniqueName: \"kubernetes.io/projected/b3231f7c-5ad1-4376-9bb2-3dec096cb340-kube-api-access-p4h8f\") pod \"prometheus-operator-68bcd7f987-2snb2\" (UID: \"b3231f7c-5ad1-4376-9bb2-3dec096cb340\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2"
Sep 11 11:50:43.527772 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.527747    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2"
Sep 11 11:50:43.617645 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.617153    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmj9n" event={"ID":"89c76d6c-b4b3-442a-8081-0ec81e5894a6","Type":"ContainerStarted","Data":"54dd5c273abbe2c987cc84a88c53510696996c26d784688d420a3394b2b96764"}
Sep 11 11:50:43.617645 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.617211    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmj9n" event={"ID":"89c76d6c-b4b3-442a-8081-0ec81e5894a6","Type":"ContainerStarted","Data":"5d40ad2fb23a6cf4b2cdfc5a662fe3005106b0be5124ebcb461cdda92a1d0427"}
Sep 11 11:50:43.617645 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.617263    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:50:43.620738 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.620715    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-859g8" event={"ID":"8592c40b-2ab2-47ee-872b-4aa604fe5b36","Type":"ContainerStarted","Data":"75bd14e847b760e389e22b4f1ed012e71c837b1646c44cbba732695b4a7373b1"}
Sep 11 11:50:43.637128 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.637087    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hmj9n" podStartSLOduration=129.999908339 podStartE2EDuration="2m11.637071944s" totalImagesPullingTime="1.637163605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:50:43.635861732 +0000 UTC m=+166.057730127" watchObservedRunningTime="2026-09-11 11:50:43.637071944 +0000 UTC m=+166.058940337"
Sep 11 11:50:43.794642 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:43.794564    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-2snb2"]
Sep 11 11:50:43.797481 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:50:43.797439    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3231f7c_5ad1_4376_9bb2_3dec096cb340.slice/crio-3fae3b99008403506d32198ce136d4c94859a3afcec15da3a51bcbdd6dc4520e WatchSource:0}: Error finding container 3fae3b99008403506d32198ce136d4c94859a3afcec15da3a51bcbdd6dc4520e: Status 404 returned error can't find the container with id 3fae3b99008403506d32198ce136d4c94859a3afcec15da3a51bcbdd6dc4520e
Sep 11 11:50:44.625599 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:44.625556    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2" event={"ID":"b3231f7c-5ad1-4376-9bb2-3dec096cb340","Type":"ContainerStarted","Data":"3fae3b99008403506d32198ce136d4c94859a3afcec15da3a51bcbdd6dc4520e"}
Sep 11 11:50:45.631034 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:45.631004    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-859g8" event={"ID":"8592c40b-2ab2-47ee-872b-4aa604fe5b36","Type":"ContainerStarted","Data":"d76189ea18005eec6e3c079717501a8d2b05b435167afcdc94dd18f9f9d7139c"}
Sep 11 11:50:45.632883 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:45.632855    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2" event={"ID":"b3231f7c-5ad1-4376-9bb2-3dec096cb340","Type":"ContainerStarted","Data":"8cfac31be110e0ce58aad3c4816020d32ad109ff4508bfa845eeb98b0734d0b4"}
Sep 11 11:50:45.632970 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:45.632908    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2" event={"ID":"b3231f7c-5ad1-4376-9bb2-3dec096cb340","Type":"ContainerStarted","Data":"7aa6d3b4d0e924aca0c351298a4007ac31bbf758263a8063d5e7f2a54b581434"}
Sep 11 11:50:45.651330 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:45.651273    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-859g8" podStartSLOduration=33.964590799 podStartE2EDuration="36.65125259s" totalImagesPullingTime="2.686661791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:50:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:50:45.648646506 +0000 UTC m=+168.070514908" watchObservedRunningTime="2026-09-11 11:50:45.65125259 +0000 UTC m=+168.073120976"
Sep 11 11:50:45.666666 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:45.666617    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-2snb2" podStartSLOduration=1.013000662 podStartE2EDuration="2.666579805s" totalImagesPullingTime="1.653579143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:50:45.666174626 +0000 UTC m=+168.088043006" watchObservedRunningTime="2026-09-11 11:50:45.666579805 +0000 UTC m=+168.088448186"
Sep 11 11:50:45.912890 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:45.912866    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:50:47.708030 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.707995    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"]
Sep 11 11:50:47.711399 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.711381    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:47.720007 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.719985    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 11 11:50:47.723282 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.723267    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 11 11:50:47.726200 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.726165    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-h5th9\""
Sep 11 11:50:47.726312 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.726188    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 11 11:50:47.773935 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.773911    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"]
Sep 11 11:50:47.863432 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.863411    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2sql7"]
Sep 11 11:50:47.866601 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.866569    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:47.869565 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.869548    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 11:50:47.869945 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.869929    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 11:50:47.870025 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.869973    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lh48f\""
Sep 11 11:50:47.870137 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.870113    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 11:50:47.874489 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.874471    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0902197e-f7cd-43ba-b12e-243318ac4fd0-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:47.874564 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.874520    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4lqm\" (UniqueName: \"kubernetes.io/projected/0902197e-f7cd-43ba-b12e-243318ac4fd0-kube-api-access-c4lqm\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:47.874622 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.874572    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0902197e-f7cd-43ba-b12e-243318ac4fd0-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:47.874660 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.874628    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0902197e-f7cd-43ba-b12e-243318ac4fd0-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:47.874660 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.874648    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0902197e-f7cd-43ba-b12e-243318ac4fd0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:47.874734 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.874680    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0902197e-f7cd-43ba-b12e-243318ac4fd0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:47.975496 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.975439    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0902197e-f7cd-43ba-b12e-243318ac4fd0-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:47.975496 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.975473    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0902197e-f7cd-43ba-b12e-243318ac4fd0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:47.975692 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.975501    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76c0c6c7-2fed-4522-8797-11bc095ce67e-metrics-client-ca\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:47.975692 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.975533    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0902197e-f7cd-43ba-b12e-243318ac4fd0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:47.975692 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.975564    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76c0c6c7-2fed-4522-8797-11bc095ce67e-sys\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:47.975692 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.975609    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76c0c6c7-2fed-4522-8797-11bc095ce67e-node-exporter-textfile\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:47.975692 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.975633    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76c0c6c7-2fed-4522-8797-11bc095ce67e-node-exporter-wtmp\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:47.975692 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.975661    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76c0c6c7-2fed-4522-8797-11bc095ce67e-root\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:47.975692 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.975681    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76c0c6c7-2fed-4522-8797-11bc095ce67e-node-exporter-accelerators-collector-config\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:47.976029 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.975736    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0902197e-f7cd-43ba-b12e-243318ac4fd0-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:47.976029 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.975794    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76c0c6c7-2fed-4522-8797-11bc095ce67e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:47.976029 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.975834    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4lqm\" (UniqueName: \"kubernetes.io/projected/0902197e-f7cd-43ba-b12e-243318ac4fd0-kube-api-access-c4lqm\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:47.976029 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.975857    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjhb8\" (UniqueName: \"kubernetes.io/projected/76c0c6c7-2fed-4522-8797-11bc095ce67e-kube-api-access-jjhb8\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:47.976029 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.975897    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0902197e-f7cd-43ba-b12e-243318ac4fd0-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:47.976029 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.975901    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0902197e-f7cd-43ba-b12e-243318ac4fd0-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:47.976029 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.975921    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76c0c6c7-2fed-4522-8797-11bc095ce67e-node-exporter-tls\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:47.976377 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.976267    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0902197e-f7cd-43ba-b12e-243318ac4fd0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:47.976682 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.976651    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0902197e-f7cd-43ba-b12e-243318ac4fd0-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:47.978062 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.978036    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0902197e-f7cd-43ba-b12e-243318ac4fd0-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:47.978062 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:47.978048    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0902197e-f7cd-43ba-b12e-243318ac4fd0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:48.001198 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.001171    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4lqm\" (UniqueName: \"kubernetes.io/projected/0902197e-f7cd-43ba-b12e-243318ac4fd0-kube-api-access-c4lqm\") pod \"kube-state-metrics-59fbdd77d-97kqm\" (UID: \"0902197e-f7cd-43ba-b12e-243318ac4fd0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:48.020064 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.020039    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"
Sep 11 11:50:48.076393 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.076358    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76c0c6c7-2fed-4522-8797-11bc095ce67e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.076507 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.076414    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjhb8\" (UniqueName: \"kubernetes.io/projected/76c0c6c7-2fed-4522-8797-11bc095ce67e-kube-api-access-jjhb8\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.076507 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.076454    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76c0c6c7-2fed-4522-8797-11bc095ce67e-node-exporter-tls\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.076507 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.076496    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76c0c6c7-2fed-4522-8797-11bc095ce67e-metrics-client-ca\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.076686 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.076535    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76c0c6c7-2fed-4522-8797-11bc095ce67e-sys\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.076686 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.076566    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76c0c6c7-2fed-4522-8797-11bc095ce67e-node-exporter-textfile\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.076686 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.076606    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76c0c6c7-2fed-4522-8797-11bc095ce67e-node-exporter-wtmp\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.076686 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.076640    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76c0c6c7-2fed-4522-8797-11bc095ce67e-root\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.076686 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.076666    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76c0c6c7-2fed-4522-8797-11bc095ce67e-node-exporter-accelerators-collector-config\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.076950 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.076704    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76c0c6c7-2fed-4522-8797-11bc095ce67e-sys\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.077419 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.077135    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76c0c6c7-2fed-4522-8797-11bc095ce67e-node-exporter-wtmp\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.077419 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.077310    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76c0c6c7-2fed-4522-8797-11bc095ce67e-node-exporter-accelerators-collector-config\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.077419 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.077382    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76c0c6c7-2fed-4522-8797-11bc095ce67e-node-exporter-textfile\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.077419 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.077379    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76c0c6c7-2fed-4522-8797-11bc095ce67e-root\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.078485 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.078462    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76c0c6c7-2fed-4522-8797-11bc095ce67e-metrics-client-ca\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.079805 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.079758    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76c0c6c7-2fed-4522-8797-11bc095ce67e-node-exporter-tls\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.080385 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.080323    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76c0c6c7-2fed-4522-8797-11bc095ce67e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.128493 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.128465    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjhb8\" (UniqueName: \"kubernetes.io/projected/76c0c6c7-2fed-4522-8797-11bc095ce67e-kube-api-access-jjhb8\") pod \"node-exporter-2sql7\" (UID: \"76c0c6c7-2fed-4522-8797-11bc095ce67e\") " pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.175486 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.175453    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2sql7"
Sep 11 11:50:48.186124 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:50:48.186104    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c0c6c7_2fed_4522_8797_11bc095ce67e.slice/crio-8546d7b2c06153cbfeda52fed06986d5ffcb74c01b6384c47ddecaeaffb7cc8a WatchSource:0}: Error finding container 8546d7b2c06153cbfeda52fed06986d5ffcb74c01b6384c47ddecaeaffb7cc8a: Status 404 returned error can't find the container with id 8546d7b2c06153cbfeda52fed06986d5ffcb74c01b6384c47ddecaeaffb7cc8a
Sep 11 11:50:48.318737 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.318719    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm"]
Sep 11 11:50:48.322188 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:50:48.322159    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0902197e_f7cd_43ba_b12e_243318ac4fd0.slice/crio-3e6c3170bf27e770508e10c92b0e66bd762e142fc1b18ae69551ec60fe25b49a WatchSource:0}: Error finding container 3e6c3170bf27e770508e10c92b0e66bd762e142fc1b18ae69551ec60fe25b49a: Status 404 returned error can't find the container with id 3e6c3170bf27e770508e10c92b0e66bd762e142fc1b18ae69551ec60fe25b49a
Sep 11 11:50:48.643879 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.643795    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2sql7" event={"ID":"76c0c6c7-2fed-4522-8797-11bc095ce67e","Type":"ContainerStarted","Data":"8546d7b2c06153cbfeda52fed06986d5ffcb74c01b6384c47ddecaeaffb7cc8a"}
Sep 11 11:50:48.644994 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:48.644963    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm" event={"ID":"0902197e-f7cd-43ba-b12e-243318ac4fd0","Type":"ContainerStarted","Data":"3e6c3170bf27e770508e10c92b0e66bd762e142fc1b18ae69551ec60fe25b49a"}
Sep 11 11:50:50.069342 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:50.069275    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:50:50.654345 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:50.654310    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2sql7" event={"ID":"76c0c6c7-2fed-4522-8797-11bc095ce67e","Type":"ContainerStarted","Data":"93468829de8f58b6d2074305c001465865a9f63abfdace607df7fb2ab003b19f"}
Sep 11 11:50:50.657327 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:50.657210    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm" event={"ID":"0902197e-f7cd-43ba-b12e-243318ac4fd0","Type":"ContainerStarted","Data":"49addeccff2ee262bb9c58c2bed82fcb5b74452e24e404c7d1629568e42ba3fb"}
Sep 11 11:50:50.657445 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:50.657365    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm" event={"ID":"0902197e-f7cd-43ba-b12e-243318ac4fd0","Type":"ContainerStarted","Data":"7e0e36bd4d810984f74c8185e96a30550605a1cd637fdd1ff7c13a6ddc1c60c5"}
Sep 11 11:50:50.657445 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:50.657398    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm" event={"ID":"0902197e-f7cd-43ba-b12e-243318ac4fd0","Type":"ContainerStarted","Data":"44a4d52fa3f9e94daa761262d47cb5881b0a2bfc40f8d14ae0ef2c67f3a037e4"}
Sep 11 11:50:50.703603 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:50.703545    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-97kqm" podStartSLOduration=2.259730188 podStartE2EDuration="3.703532348s" totalImagesPullingTime="1.44380216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:50:50.703266219 +0000 UTC m=+173.125134619" watchObservedRunningTime="2026-09-11 11:50:50.703532348 +0000 UTC m=+173.125400727"
Sep 11 11:50:51.662523 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:51.662486    2569 generic.go:356] "Generic (PLEG): container finished" podID="76c0c6c7-2fed-4522-8797-11bc095ce67e" containerID="93468829de8f58b6d2074305c001465865a9f63abfdace607df7fb2ab003b19f" exitCode=0
Sep 11 11:50:51.663013 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:51.662579    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2sql7" event={"ID":"76c0c6c7-2fed-4522-8797-11bc095ce67e","Type":"ContainerDied","Data":"93468829de8f58b6d2074305c001465865a9f63abfdace607df7fb2ab003b19f"}
Sep 11 11:50:52.401192 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.401154    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-wbfgd"]
Sep 11 11:50:52.404549 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.404529    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wbfgd"
Sep 11 11:50:52.406661 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.406633    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 11:50:52.406829 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.406809    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nrcmb\""
Sep 11 11:50:52.407500 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.407479    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h8jh\" (UniqueName: \"kubernetes.io/projected/b82fa9c1-56f0-4167-a128-dc6948db6614-kube-api-access-2h8jh\") pod \"monitoring-plugin-5d85778b9d-wbfgd\" (UID: \"b82fa9c1-56f0-4167-a128-dc6948db6614\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wbfgd"
Sep 11 11:50:52.407597 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.407537    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b82fa9c1-56f0-4167-a128-dc6948db6614-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-wbfgd\" (UID: \"b82fa9c1-56f0-4167-a128-dc6948db6614\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wbfgd"
Sep 11 11:50:52.413440 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.413417    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-wbfgd"]
Sep 11 11:50:52.508270 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.508245    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h8jh\" (UniqueName: \"kubernetes.io/projected/b82fa9c1-56f0-4167-a128-dc6948db6614-kube-api-access-2h8jh\") pod \"monitoring-plugin-5d85778b9d-wbfgd\" (UID: \"b82fa9c1-56f0-4167-a128-dc6948db6614\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wbfgd"
Sep 11 11:50:52.508376 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.508279    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b82fa9c1-56f0-4167-a128-dc6948db6614-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-wbfgd\" (UID: \"b82fa9c1-56f0-4167-a128-dc6948db6614\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wbfgd"
Sep 11 11:50:52.508735 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:52.508701    2569 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 11 11:50:52.508833 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:50:52.508798    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b82fa9c1-56f0-4167-a128-dc6948db6614-monitoring-plugin-cert podName:b82fa9c1-56f0-4167-a128-dc6948db6614 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:53.008767302 +0000 UTC m=+175.430635723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/b82fa9c1-56f0-4167-a128-dc6948db6614-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-wbfgd" (UID: "b82fa9c1-56f0-4167-a128-dc6948db6614") : secret "monitoring-plugin-cert" not found
Sep 11 11:50:52.518348 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.518318    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h8jh\" (UniqueName: \"kubernetes.io/projected/b82fa9c1-56f0-4167-a128-dc6948db6614-kube-api-access-2h8jh\") pod \"monitoring-plugin-5d85778b9d-wbfgd\" (UID: \"b82fa9c1-56f0-4167-a128-dc6948db6614\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wbfgd"
Sep 11 11:50:52.668117 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.668060    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2sql7" event={"ID":"76c0c6c7-2fed-4522-8797-11bc095ce67e","Type":"ContainerStarted","Data":"62fc792a1e4201e007908fd0ae6b4d098bb3dcd61c5d8a1e07ef8fd928a90cca"}
Sep 11 11:50:52.668409 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.668127    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2sql7" event={"ID":"76c0c6c7-2fed-4522-8797-11bc095ce67e","Type":"ContainerStarted","Data":"b99efea1a9cbc11068f093416c81926bd0c89a6bb7f2cd24d0fd5ca795b94045"}
Sep 11 11:50:52.691159 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.691119    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2sql7" podStartSLOduration=3.289620296 podStartE2EDuration="5.691096899s" totalImagesPullingTime="2.33489738s" totalInitContainerRuntime="66.579223ms" isStatefulPod=true podCreationTimestamp="2026-09-11 11:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:50:52.689517828 +0000 UTC m=+175.111386231" watchObservedRunningTime="2026-09-11 11:50:52.691096899 +0000 UTC m=+175.112965270"
Sep 11 11:50:52.953529 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.953442    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7669fb6996-656pl"]
Sep 11 11:50:52.957084 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.957060    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:52.960168 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.960143    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 11:50:52.960296 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.960147    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 11:50:52.960296 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.960188    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 11:50:52.961005 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.960985    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 11:50:52.961201 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.961187    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 11:50:52.966619 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.965336    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zdxn2\""
Sep 11 11:50:52.970406 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.970387    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 11:50:52.978857 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:52.978837    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7669fb6996-656pl"]
Sep 11 11:50:53.012531 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.012507    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbff4b49-3525-46aa-a15c-be05c0e439cc-metrics-client-ca\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.012634 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.012532    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbff4b49-3525-46aa-a15c-be05c0e439cc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.012634 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.012554    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dbff4b49-3525-46aa-a15c-be05c0e439cc-telemeter-client-tls\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.012634 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.012571    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dbff4b49-3525-46aa-a15c-be05c0e439cc-secret-telemeter-client\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.012634 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.012624    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b82fa9c1-56f0-4167-a128-dc6948db6614-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-wbfgd\" (UID: \"b82fa9c1-56f0-4167-a128-dc6948db6614\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wbfgd"
Sep 11 11:50:53.012833 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.012675    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8rhc\" (UniqueName: \"kubernetes.io/projected/dbff4b49-3525-46aa-a15c-be05c0e439cc-kube-api-access-v8rhc\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.012833 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.012696    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbff4b49-3525-46aa-a15c-be05c0e439cc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.012833 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.012720    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbff4b49-3525-46aa-a15c-be05c0e439cc-serving-certs-ca-bundle\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.012833 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.012743    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dbff4b49-3525-46aa-a15c-be05c0e439cc-federate-client-tls\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.014926 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.014906    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b82fa9c1-56f0-4167-a128-dc6948db6614-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-wbfgd\" (UID: \"b82fa9c1-56f0-4167-a128-dc6948db6614\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wbfgd"
Sep 11 11:50:53.015007 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.014986    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wbfgd"
Sep 11 11:50:53.113342 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.113308    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8rhc\" (UniqueName: \"kubernetes.io/projected/dbff4b49-3525-46aa-a15c-be05c0e439cc-kube-api-access-v8rhc\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.113492 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.113353    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbff4b49-3525-46aa-a15c-be05c0e439cc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.113492 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.113391    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbff4b49-3525-46aa-a15c-be05c0e439cc-serving-certs-ca-bundle\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.113492 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.113426    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dbff4b49-3525-46aa-a15c-be05c0e439cc-federate-client-tls\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.113492 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.113462    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbff4b49-3525-46aa-a15c-be05c0e439cc-metrics-client-ca\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.113492 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.113488    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbff4b49-3525-46aa-a15c-be05c0e439cc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.113768 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.113725    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dbff4b49-3525-46aa-a15c-be05c0e439cc-telemeter-client-tls\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.113820 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.113767    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dbff4b49-3525-46aa-a15c-be05c0e439cc-secret-telemeter-client\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.114285 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.114236    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbff4b49-3525-46aa-a15c-be05c0e439cc-serving-certs-ca-bundle\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.114390 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.114318    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbff4b49-3525-46aa-a15c-be05c0e439cc-metrics-client-ca\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.114390 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.114370    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbff4b49-3525-46aa-a15c-be05c0e439cc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.116383 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.116362    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbff4b49-3525-46aa-a15c-be05c0e439cc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.116609 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.116573    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dbff4b49-3525-46aa-a15c-be05c0e439cc-telemeter-client-tls\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.116670 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.116577    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dbff4b49-3525-46aa-a15c-be05c0e439cc-secret-telemeter-client\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.116670 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.116621    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dbff4b49-3525-46aa-a15c-be05c0e439cc-federate-client-tls\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.122102 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.122079    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8rhc\" (UniqueName: \"kubernetes.io/projected/dbff4b49-3525-46aa-a15c-be05c0e439cc-kube-api-access-v8rhc\") pod \"telemeter-client-7669fb6996-656pl\" (UID: \"dbff4b49-3525-46aa-a15c-be05c0e439cc\") " pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.270243 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.270223    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7669fb6996-656pl"
Sep 11 11:50:53.289342 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.289321    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-wbfgd"]
Sep 11 11:50:53.290653 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:50:53.290629    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82fa9c1_56f0_4167_a128_dc6948db6614.slice/crio-d9928f4d3d06e236b82139ccd480d296c95a4603c3ac41c3898c63456ca2e2c7 WatchSource:0}: Error finding container d9928f4d3d06e236b82139ccd480d296c95a4603c3ac41c3898c63456ca2e2c7: Status 404 returned error can't find the container with id d9928f4d3d06e236b82139ccd480d296c95a4603c3ac41c3898c63456ca2e2c7
Sep 11 11:50:53.557982 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.557911    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7669fb6996-656pl"]
Sep 11 11:50:53.561147 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:50:53.561122    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbff4b49_3525_46aa_a15c_be05c0e439cc.slice/crio-e171af118175c6eca7816afad977f7137d313f755aec03bc60b13a1e87f73f0b WatchSource:0}: Error finding container e171af118175c6eca7816afad977f7137d313f755aec03bc60b13a1e87f73f0b: Status 404 returned error can't find the container with id e171af118175c6eca7816afad977f7137d313f755aec03bc60b13a1e87f73f0b
Sep 11 11:50:53.672825 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.672792    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wbfgd" event={"ID":"b82fa9c1-56f0-4167-a128-dc6948db6614","Type":"ContainerStarted","Data":"d9928f4d3d06e236b82139ccd480d296c95a4603c3ac41c3898c63456ca2e2c7"}
Sep 11 11:50:53.674107 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:53.674083    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7669fb6996-656pl" event={"ID":"dbff4b49-3525-46aa-a15c-be05c0e439cc","Type":"ContainerStarted","Data":"e171af118175c6eca7816afad977f7137d313f755aec03bc60b13a1e87f73f0b"}
Sep 11 11:50:54.635176 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:54.635145    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hmj9n"
Sep 11 11:50:55.683743 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:55.683714    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wbfgd" event={"ID":"b82fa9c1-56f0-4167-a128-dc6948db6614","Type":"ContainerStarted","Data":"899a37cdd567d0bfd6fd416458ed8bbe2bb6131544f3502c1b20e41781f00d2a"}
Sep 11 11:50:55.684112 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:55.683802    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wbfgd"
Sep 11 11:50:55.685395 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:55.685367    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7669fb6996-656pl" event={"ID":"dbff4b49-3525-46aa-a15c-be05c0e439cc","Type":"ContainerStarted","Data":"f4f21fbe148ba48b8016a3626b866958f257ea7c559d8ad32738dc15e57a094b"}
Sep 11 11:50:55.688757 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:55.688732    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wbfgd"
Sep 11 11:50:55.702593 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:55.702552    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wbfgd" podStartSLOduration=1.441519352 podStartE2EDuration="3.70253987s" totalImagesPullingTime="2.261020518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:50:55.700998482 +0000 UTC m=+178.122866862" watchObservedRunningTime="2026-09-11 11:50:55.70253987 +0000 UTC m=+178.124408247"
Sep 11 11:50:56.690897 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:56.690860    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7669fb6996-656pl" event={"ID":"dbff4b49-3525-46aa-a15c-be05c0e439cc","Type":"ContainerStarted","Data":"7ef1da6b7b64a7679fa3d3df4a943623c418a1508cc4227cc59d9f07f3b90a09"}
Sep 11 11:50:56.691261 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:56.690921    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7669fb6996-656pl" event={"ID":"dbff4b49-3525-46aa-a15c-be05c0e439cc","Type":"ContainerStarted","Data":"59b6a82f3e78f04ced74e4c6098169a113f672e1aa640a1d6f5cc068c52a7c1d"}
Sep 11 11:50:56.729064 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:56.728963    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7669fb6996-656pl" podStartSLOduration=1.892613596 podStartE2EDuration="4.728944357s" totalImagesPullingTime="2.836330761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:50:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:50:56.727560405 +0000 UTC m=+179.149428786" watchObservedRunningTime="2026-09-11 11:50:56.728944357 +0000 UTC m=+179.150812738"
Sep 11 11:50:58.593890 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:50:58.593862    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c869f9486-nt2v4"
Sep 11 11:51:00.926323 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:00.926279    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69b567967-lrbqw" podUID="c7448b21-ef18-4ab5-bea4-cebb5d20fb13" containerName="registry" containerID="cri-o://ed43f7255310d269c8141bc7ca38014019b4dab1390ef5930fbbd0375e442c0c" gracePeriod=30
Sep 11 11:51:01.152478 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.152456    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:51:01.170544 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.170519    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx2fd\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-kube-api-access-kx2fd\") pod \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") "
Sep 11 11:51:01.170641 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.170605    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-bound-sa-token\") pod \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") "
Sep 11 11:51:01.170641 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.170636    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-certificates\") pod \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") "
Sep 11 11:51:01.170725 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.170669    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls\") pod \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") "
Sep 11 11:51:01.170909 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.170827    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-image-registry-private-configuration\") pod \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") "
Sep 11 11:51:01.170909 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.170887    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-installation-pull-secrets\") pod \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") "
Sep 11 11:51:01.171077 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.170915    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-trusted-ca\") pod \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") "
Sep 11 11:51:01.171077 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.170963    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-ca-trust-extracted\") pod \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\" (UID: \"c7448b21-ef18-4ab5-bea4-cebb5d20fb13\") "
Sep 11 11:51:01.171077 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.171060    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c7448b21-ef18-4ab5-bea4-cebb5d20fb13" (UID: "c7448b21-ef18-4ab5-bea4-cebb5d20fb13"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 11:51:01.171355 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.171331    2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-certificates\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:51:01.171630 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.171601    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c7448b21-ef18-4ab5-bea4-cebb5d20fb13" (UID: "c7448b21-ef18-4ab5-bea4-cebb5d20fb13"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 11:51:01.173280 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.173249    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c7448b21-ef18-4ab5-bea4-cebb5d20fb13" (UID: "c7448b21-ef18-4ab5-bea4-cebb5d20fb13"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:51:01.173405 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.173302    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c7448b21-ef18-4ab5-bea4-cebb5d20fb13" (UID: "c7448b21-ef18-4ab5-bea4-cebb5d20fb13"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:51:01.173736 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.173714    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-kube-api-access-kx2fd" (OuterVolumeSpecName: "kube-api-access-kx2fd") pod "c7448b21-ef18-4ab5-bea4-cebb5d20fb13" (UID: "c7448b21-ef18-4ab5-bea4-cebb5d20fb13"). InnerVolumeSpecName "kube-api-access-kx2fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:51:01.173939 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.173916    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c7448b21-ef18-4ab5-bea4-cebb5d20fb13" (UID: "c7448b21-ef18-4ab5-bea4-cebb5d20fb13"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 11:51:01.174008 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.173960    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c7448b21-ef18-4ab5-bea4-cebb5d20fb13" (UID: "c7448b21-ef18-4ab5-bea4-cebb5d20fb13"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 11:51:01.181169 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.181138    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c7448b21-ef18-4ab5-bea4-cebb5d20fb13" (UID: "c7448b21-ef18-4ab5-bea4-cebb5d20fb13"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:51:01.272116 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.272082    2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-bound-sa-token\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:51:01.272116 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.272113    2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-registry-tls\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:51:01.272116 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.272123    2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-image-registry-private-configuration\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:51:01.272284 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.272132    2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-installation-pull-secrets\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:51:01.272284 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.272140    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-trusted-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:51:01.272284 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.272149    2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-ca-trust-extracted\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:51:01.272284 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.272157    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx2fd\" (UniqueName: \"kubernetes.io/projected/c7448b21-ef18-4ab5-bea4-cebb5d20fb13-kube-api-access-kx2fd\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:51:01.712027 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.712002    2569 generic.go:356] "Generic (PLEG): container finished" podID="c7448b21-ef18-4ab5-bea4-cebb5d20fb13" containerID="ed43f7255310d269c8141bc7ca38014019b4dab1390ef5930fbbd0375e442c0c" exitCode=0
Sep 11 11:51:01.712173 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.712048    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69b567967-lrbqw"
Sep 11 11:51:01.712173 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.712073    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69b567967-lrbqw" event={"ID":"c7448b21-ef18-4ab5-bea4-cebb5d20fb13","Type":"ContainerDied","Data":"ed43f7255310d269c8141bc7ca38014019b4dab1390ef5930fbbd0375e442c0c"}
Sep 11 11:51:01.712173 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.712110    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69b567967-lrbqw" event={"ID":"c7448b21-ef18-4ab5-bea4-cebb5d20fb13","Type":"ContainerDied","Data":"5cbf0e730e4d75ea354b40cb16f0a650700d9b2ae4aa5263f1ee608075102a2a"}
Sep 11 11:51:01.712173 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.712130    2569 scope.go:117] "RemoveContainer" containerID="ed43f7255310d269c8141bc7ca38014019b4dab1390ef5930fbbd0375e442c0c"
Sep 11 11:51:01.720441 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.720421    2569 scope.go:117] "RemoveContainer" containerID="ed43f7255310d269c8141bc7ca38014019b4dab1390ef5930fbbd0375e442c0c"
Sep 11 11:51:01.720729 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:51:01.720710    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed43f7255310d269c8141bc7ca38014019b4dab1390ef5930fbbd0375e442c0c\": container with ID starting with ed43f7255310d269c8141bc7ca38014019b4dab1390ef5930fbbd0375e442c0c not found: ID does not exist" containerID="ed43f7255310d269c8141bc7ca38014019b4dab1390ef5930fbbd0375e442c0c"
Sep 11 11:51:01.720796 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.720736    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed43f7255310d269c8141bc7ca38014019b4dab1390ef5930fbbd0375e442c0c"} err="failed to get container status \"ed43f7255310d269c8141bc7ca38014019b4dab1390ef5930fbbd0375e442c0c\": rpc error: code = NotFound desc = could not find container \"ed43f7255310d269c8141bc7ca38014019b4dab1390ef5930fbbd0375e442c0c\": container with ID starting with ed43f7255310d269c8141bc7ca38014019b4dab1390ef5930fbbd0375e442c0c not found: ID does not exist"
Sep 11 11:51:01.732413 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.732391    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69b567967-lrbqw"]
Sep 11 11:51:01.739494 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.739471    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69b567967-lrbqw"]
Sep 11 11:51:01.873649 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.873627    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-jb96z"]
Sep 11 11:51:01.873889 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.873877    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7448b21-ef18-4ab5-bea4-cebb5d20fb13" containerName="registry"
Sep 11 11:51:01.873937 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.873890    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7448b21-ef18-4ab5-bea4-cebb5d20fb13" containerName="registry"
Sep 11 11:51:01.873971 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.873943    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7448b21-ef18-4ab5-bea4-cebb5d20fb13" containerName="registry"
Sep 11 11:51:01.876521 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.876506    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-jb96z"
Sep 11 11:51:01.879301 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.879274    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 11:51:01.880054 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.879790    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zs4nt\""
Sep 11 11:51:01.880054 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.879994    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 11:51:01.889155 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.889136    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-jb96z"]
Sep 11 11:51:01.978058 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:01.977997    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88tgl\" (UniqueName: \"kubernetes.io/projected/efb5b029-a20f-4b03-8814-4069153d2906-kube-api-access-88tgl\") pod \"downloads-76f99584f4-jb96z\" (UID: \"efb5b029-a20f-4b03-8814-4069153d2906\") " pod="openshift-console/downloads-76f99584f4-jb96z"
Sep 11 11:51:02.068631 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:02.068602    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7448b21-ef18-4ab5-bea4-cebb5d20fb13" path="/var/lib/kubelet/pods/c7448b21-ef18-4ab5-bea4-cebb5d20fb13/volumes"
Sep 11 11:51:02.079126 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:02.079100    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88tgl\" (UniqueName: \"kubernetes.io/projected/efb5b029-a20f-4b03-8814-4069153d2906-kube-api-access-88tgl\") pod \"downloads-76f99584f4-jb96z\" (UID: \"efb5b029-a20f-4b03-8814-4069153d2906\") " pod="openshift-console/downloads-76f99584f4-jb96z"
Sep 11 11:51:02.088229 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:02.088200    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88tgl\" (UniqueName: \"kubernetes.io/projected/efb5b029-a20f-4b03-8814-4069153d2906-kube-api-access-88tgl\") pod \"downloads-76f99584f4-jb96z\" (UID: \"efb5b029-a20f-4b03-8814-4069153d2906\") " pod="openshift-console/downloads-76f99584f4-jb96z"
Sep 11 11:51:02.190416 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:02.190384    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-jb96z"
Sep 11 11:51:02.460651 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:02.460630    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-jb96z"]
Sep 11 11:51:02.462975 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:51:02.462950    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb5b029_a20f_4b03_8814_4069153d2906.slice/crio-b6913bfc0cba2f94cafba9a904e8686fb8308db70ab88062c76a2bf385503e8c WatchSource:0}: Error finding container b6913bfc0cba2f94cafba9a904e8686fb8308db70ab88062c76a2bf385503e8c: Status 404 returned error can't find the container with id b6913bfc0cba2f94cafba9a904e8686fb8308db70ab88062c76a2bf385503e8c
Sep 11 11:51:02.717845 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:02.717819    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-jb96z" event={"ID":"efb5b029-a20f-4b03-8814-4069153d2906","Type":"ContainerStarted","Data":"b6913bfc0cba2f94cafba9a904e8686fb8308db70ab88062c76a2bf385503e8c"}
Sep 11 11:51:14.091785 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.091753    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55f7bf4994-2cn9s"]
Sep 11 11:51:14.096564 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.096534    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.099385 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.099359    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 11:51:14.099977 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.099957    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2p7cs\""
Sep 11 11:51:14.100342 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.100279    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 11:51:14.100342 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.100296    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 11:51:14.100342 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.100321    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 11:51:14.100730 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.100711    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 11:51:14.108193 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.108142    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55f7bf4994-2cn9s"]
Sep 11 11:51:14.181263 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.181225    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-console-config\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.181410 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.181287    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-service-ca\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.181410 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.181326    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-oauth-serving-cert\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.181410 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.181384    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d2c2\" (UniqueName: \"kubernetes.io/projected/7be71309-2439-482d-9202-9effbfb8fce3-kube-api-access-2d2c2\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.181562 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.181422    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7be71309-2439-482d-9202-9effbfb8fce3-console-oauth-config\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.181562 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.181463    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7be71309-2439-482d-9202-9effbfb8fce3-console-serving-cert\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.282135 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.282105    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-console-config\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.282310 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.282152    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-service-ca\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.282310 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.282184    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-oauth-serving-cert\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.282310 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.282212    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d2c2\" (UniqueName: \"kubernetes.io/projected/7be71309-2439-482d-9202-9effbfb8fce3-kube-api-access-2d2c2\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.282310 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.282239    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7be71309-2439-482d-9202-9effbfb8fce3-console-oauth-config\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.282310 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.282274    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7be71309-2439-482d-9202-9effbfb8fce3-console-serving-cert\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.283313 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.283282    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-oauth-serving-cert\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.283436 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.282823    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-console-config\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.284782 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.283518    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-service-ca\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.288872 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.286455    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7be71309-2439-482d-9202-9effbfb8fce3-console-oauth-config\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.289768 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.289743    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7be71309-2439-482d-9202-9effbfb8fce3-console-serving-cert\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.292206 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.292181    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d2c2\" (UniqueName: \"kubernetes.io/projected/7be71309-2439-482d-9202-9effbfb8fce3-kube-api-access-2d2c2\") pod \"console-55f7bf4994-2cn9s\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") " pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:14.408886 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:14.408818    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:18.489837 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:18.489812    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55f7bf4994-2cn9s"]
Sep 11 11:51:18.491225 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:51:18.491194    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be71309_2439_482d_9202_9effbfb8fce3.slice/crio-1911e5d817f640ff6aef6213459152402a9d6af7aca181cc7ad2dd6cbf85a5cb WatchSource:0}: Error finding container 1911e5d817f640ff6aef6213459152402a9d6af7aca181cc7ad2dd6cbf85a5cb: Status 404 returned error can't find the container with id 1911e5d817f640ff6aef6213459152402a9d6af7aca181cc7ad2dd6cbf85a5cb
Sep 11 11:51:18.788793 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:18.788756    2569 generic.go:356] "Generic (PLEG): container finished" podID="236f4ed7-e91c-49de-8c80-8fb3490942c4" containerID="d3b65d447dad0ad61269ac2b09251818bd6426c776b222f147a5c307b919b5c1" exitCode=0
Sep 11 11:51:18.788972 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:18.788801    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd" event={"ID":"236f4ed7-e91c-49de-8c80-8fb3490942c4","Type":"ContainerDied","Data":"d3b65d447dad0ad61269ac2b09251818bd6426c776b222f147a5c307b919b5c1"}
Sep 11 11:51:18.789248 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:18.789225    2569 scope.go:117] "RemoveContainer" containerID="d3b65d447dad0ad61269ac2b09251818bd6426c776b222f147a5c307b919b5c1"
Sep 11 11:51:18.791033 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:18.790916    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-jb96z" event={"ID":"efb5b029-a20f-4b03-8814-4069153d2906","Type":"ContainerStarted","Data":"86d7c01f60f4959cdba1d1f2547c8fb44a37ff6bd322cfc86b911374866f1cf2"}
Sep 11 11:51:18.791148 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:18.791056    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-jb96z"
Sep 11 11:51:18.792519 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:18.792377    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f7bf4994-2cn9s" event={"ID":"7be71309-2439-482d-9202-9effbfb8fce3","Type":"ContainerStarted","Data":"1911e5d817f640ff6aef6213459152402a9d6af7aca181cc7ad2dd6cbf85a5cb"}
Sep 11 11:51:18.797974 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:18.797952    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-jb96z"
Sep 11 11:51:18.836492 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:18.836409    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-jb96z" podStartSLOduration=1.9890101040000001 podStartE2EDuration="17.83638795s" totalImagesPullingTime="15.847377846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:51:18.83513145 +0000 UTC m=+201.256999941" watchObservedRunningTime="2026-09-11 11:51:18.83638795 +0000 UTC m=+201.258256339"
Sep 11 11:51:19.799930 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:19.799892    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wk5vd" event={"ID":"236f4ed7-e91c-49de-8c80-8fb3490942c4","Type":"ContainerStarted","Data":"a9d1f1a444e141bf5a719703415d000f5cbde2c254a585d3360ff01ed01d9666"}
Sep 11 11:51:22.609002 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.608969    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-799859f498-kzw4d"]
Sep 11 11:51:22.667372 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.667330    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-799859f498-kzw4d"]
Sep 11 11:51:22.667558 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.667480    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.677171 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.676812    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 11:51:22.761158 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.761131    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-oauth-config\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.761292 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.761180    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-serving-cert\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.761292 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.761214    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzb2q\" (UniqueName: \"kubernetes.io/projected/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-kube-api-access-vzb2q\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.761292 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.761269    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-config\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.761432 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.761338    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-oauth-serving-cert\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.761432 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.761394    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-service-ca\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.761432 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.761428    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-trusted-ca-bundle\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.862644 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.862554    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-oauth-config\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.862644 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.862627    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-serving-cert\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.862865 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.862660    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzb2q\" (UniqueName: \"kubernetes.io/projected/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-kube-api-access-vzb2q\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.862865 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.862685    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-config\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.862865 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.862728    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-oauth-serving-cert\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.862865 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.862783    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-service-ca\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.862865 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.862829    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-trusted-ca-bundle\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.863662 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.863573    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-oauth-serving-cert\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.863662 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.863644    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-config\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.863840 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.863737    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-service-ca\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.863840 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.863742    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-trusted-ca-bundle\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.865634 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.865613    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-oauth-config\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.865805 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.865788    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-serving-cert\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.872932 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.872860    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzb2q\" (UniqueName: \"kubernetes.io/projected/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-kube-api-access-vzb2q\") pod \"console-799859f498-kzw4d\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") " pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:22.980649 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:22.980618    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:23.353008 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:23.352979    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-799859f498-kzw4d"]
Sep 11 11:51:23.355890 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:51:23.355862    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712bcd5d_48fa_4b5d_98c2_f78b60a788fb.slice/crio-7893f2ed53220073237d428ce2cd3b80ece460d811ba73d0c5272c294504fd95 WatchSource:0}: Error finding container 7893f2ed53220073237d428ce2cd3b80ece460d811ba73d0c5272c294504fd95: Status 404 returned error can't find the container with id 7893f2ed53220073237d428ce2cd3b80ece460d811ba73d0c5272c294504fd95
Sep 11 11:51:23.822182 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:23.822143    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f7bf4994-2cn9s" event={"ID":"7be71309-2439-482d-9202-9effbfb8fce3","Type":"ContainerStarted","Data":"9a8698c5da160a9e39f5ce556b1362a315d43bffe38428b0737ad1cf88ff37a8"}
Sep 11 11:51:23.826797 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:23.826765    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799859f498-kzw4d" event={"ID":"712bcd5d-48fa-4b5d-98c2-f78b60a788fb","Type":"ContainerStarted","Data":"a9a7e096c73438e0058f27994ea01c77503f59af3522c4bc632fd9c73d343957"}
Sep 11 11:51:23.826922 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:23.826812    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799859f498-kzw4d" event={"ID":"712bcd5d-48fa-4b5d-98c2-f78b60a788fb","Type":"ContainerStarted","Data":"7893f2ed53220073237d428ce2cd3b80ece460d811ba73d0c5272c294504fd95"}
Sep 11 11:51:23.848599 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:23.843691    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55f7bf4994-2cn9s" podStartSLOduration=5.255992089 podStartE2EDuration="9.84366665s" totalImagesPullingTime="4.587674561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:51:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:51:23.841434493 +0000 UTC m=+206.263302895" watchObservedRunningTime="2026-09-11 11:51:23.84366665 +0000 UTC m=+206.265535031"
Sep 11 11:51:23.862579 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:23.862532    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-799859f498-kzw4d" podStartSLOduration=1.862514088 podStartE2EDuration="1.862514088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:51:23.860769293 +0000 UTC m=+206.282637695" watchObservedRunningTime="2026-09-11 11:51:23.862514088 +0000 UTC m=+206.284382468"
Sep 11 11:51:24.409432 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:24.409392    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:24.409736 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:24.409458    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:24.414881 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:24.414855    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:24.835549 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:24.835520    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:32.981399 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:32.981366    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:32.981951 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:32.981438    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:32.986111 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:32.986088    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:33.872114 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:33.872084    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:51:33.931690 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:33.931660    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55f7bf4994-2cn9s"]
Sep 11 11:51:58.951446 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:58.951405    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55f7bf4994-2cn9s" podUID="7be71309-2439-482d-9202-9effbfb8fce3" containerName="console" containerID="cri-o://9a8698c5da160a9e39f5ce556b1362a315d43bffe38428b0737ad1cf88ff37a8" gracePeriod=15
Sep 11 11:51:59.221920 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.221862    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:59.239312 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.239290    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-console-config\") pod \"7be71309-2439-482d-9202-9effbfb8fce3\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") "
Sep 11 11:51:59.239424 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.239326    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7be71309-2439-482d-9202-9effbfb8fce3-console-serving-cert\") pod \"7be71309-2439-482d-9202-9effbfb8fce3\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") "
Sep 11 11:51:59.239424 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.239347    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7be71309-2439-482d-9202-9effbfb8fce3-console-oauth-config\") pod \"7be71309-2439-482d-9202-9effbfb8fce3\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") "
Sep 11 11:51:59.239424 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.239372    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2d2c2\" (UniqueName: \"kubernetes.io/projected/7be71309-2439-482d-9202-9effbfb8fce3-kube-api-access-2d2c2\") pod \"7be71309-2439-482d-9202-9effbfb8fce3\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") "
Sep 11 11:51:59.239424 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.239393    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-service-ca\") pod \"7be71309-2439-482d-9202-9effbfb8fce3\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") "
Sep 11 11:51:59.239424 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.239425    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-oauth-serving-cert\") pod \"7be71309-2439-482d-9202-9effbfb8fce3\" (UID: \"7be71309-2439-482d-9202-9effbfb8fce3\") "
Sep 11 11:51:59.239799 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.239771    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-console-config" (OuterVolumeSpecName: "console-config") pod "7be71309-2439-482d-9202-9effbfb8fce3" (UID: "7be71309-2439-482d-9202-9effbfb8fce3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 11:51:59.239878 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.239822    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-service-ca" (OuterVolumeSpecName: "service-ca") pod "7be71309-2439-482d-9202-9effbfb8fce3" (UID: "7be71309-2439-482d-9202-9effbfb8fce3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 11:51:59.240111 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.239970    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7be71309-2439-482d-9202-9effbfb8fce3" (UID: "7be71309-2439-482d-9202-9effbfb8fce3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 11:51:59.241935 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.241895    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7be71309-2439-482d-9202-9effbfb8fce3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7be71309-2439-482d-9202-9effbfb8fce3" (UID: "7be71309-2439-482d-9202-9effbfb8fce3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 11:51:59.241935 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.241924    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be71309-2439-482d-9202-9effbfb8fce3-kube-api-access-2d2c2" (OuterVolumeSpecName: "kube-api-access-2d2c2") pod "7be71309-2439-482d-9202-9effbfb8fce3" (UID: "7be71309-2439-482d-9202-9effbfb8fce3"). InnerVolumeSpecName "kube-api-access-2d2c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:51:59.242217 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.242191    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7be71309-2439-482d-9202-9effbfb8fce3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7be71309-2439-482d-9202-9effbfb8fce3" (UID: "7be71309-2439-482d-9202-9effbfb8fce3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 11:51:59.340267 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.340236    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-console-config\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:51:59.340267 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.340260    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7be71309-2439-482d-9202-9effbfb8fce3-console-serving-cert\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:51:59.340420 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.340275    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7be71309-2439-482d-9202-9effbfb8fce3-console-oauth-config\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:51:59.340420 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.340287    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d2c2\" (UniqueName: \"kubernetes.io/projected/7be71309-2439-482d-9202-9effbfb8fce3-kube-api-access-2d2c2\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:51:59.340420 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.340300    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:51:59.340420 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.340311    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7be71309-2439-482d-9202-9effbfb8fce3-oauth-serving-cert\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:51:59.969155 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.969119    2569 generic.go:356] "Generic (PLEG): container finished" podID="7be71309-2439-482d-9202-9effbfb8fce3" containerID="9a8698c5da160a9e39f5ce556b1362a315d43bffe38428b0737ad1cf88ff37a8" exitCode=0
Sep 11 11:51:59.969604 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.969169    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f7bf4994-2cn9s"
Sep 11 11:51:59.969604 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.969199    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f7bf4994-2cn9s" event={"ID":"7be71309-2439-482d-9202-9effbfb8fce3","Type":"ContainerDied","Data":"9a8698c5da160a9e39f5ce556b1362a315d43bffe38428b0737ad1cf88ff37a8"}
Sep 11 11:51:59.969604 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.969264    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f7bf4994-2cn9s" event={"ID":"7be71309-2439-482d-9202-9effbfb8fce3","Type":"ContainerDied","Data":"1911e5d817f640ff6aef6213459152402a9d6af7aca181cc7ad2dd6cbf85a5cb"}
Sep 11 11:51:59.969604 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.969297    2569 scope.go:117] "RemoveContainer" containerID="9a8698c5da160a9e39f5ce556b1362a315d43bffe38428b0737ad1cf88ff37a8"
Sep 11 11:51:59.979338 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.979268    2569 scope.go:117] "RemoveContainer" containerID="9a8698c5da160a9e39f5ce556b1362a315d43bffe38428b0737ad1cf88ff37a8"
Sep 11 11:51:59.979648 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:51:59.979618    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a8698c5da160a9e39f5ce556b1362a315d43bffe38428b0737ad1cf88ff37a8\": container with ID starting with 9a8698c5da160a9e39f5ce556b1362a315d43bffe38428b0737ad1cf88ff37a8 not found: ID does not exist" containerID="9a8698c5da160a9e39f5ce556b1362a315d43bffe38428b0737ad1cf88ff37a8"
Sep 11 11:51:59.979740 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.979658    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a8698c5da160a9e39f5ce556b1362a315d43bffe38428b0737ad1cf88ff37a8"} err="failed to get container status \"9a8698c5da160a9e39f5ce556b1362a315d43bffe38428b0737ad1cf88ff37a8\": rpc error: code = NotFound desc = could not find container \"9a8698c5da160a9e39f5ce556b1362a315d43bffe38428b0737ad1cf88ff37a8\": container with ID starting with 9a8698c5da160a9e39f5ce556b1362a315d43bffe38428b0737ad1cf88ff37a8 not found: ID does not exist"
Sep 11 11:51:59.992888 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.992864    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55f7bf4994-2cn9s"]
Sep 11 11:51:59.998866 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:51:59.998847    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55f7bf4994-2cn9s"]
Sep 11 11:52:00.068007 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:00.067983    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7be71309-2439-482d-9202-9effbfb8fce3" path="/var/lib/kubelet/pods/7be71309-2439-482d-9202-9effbfb8fce3/volumes"
Sep 11 11:52:09.007301 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.007272    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs\") pod \"network-metrics-daemon-2xwtg\" (UID: \"ca51005b-bb90-4eac-8bc7-c64bb5463c3f\") " pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:52:09.009525 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.009506    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca51005b-bb90-4eac-8bc7-c64bb5463c3f-metrics-certs\") pod \"network-metrics-daemon-2xwtg\" (UID: \"ca51005b-bb90-4eac-8bc7-c64bb5463c3f\") " pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:52:09.273166 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.273103    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5sht6\""
Sep 11 11:52:09.280377 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.280358    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xwtg"
Sep 11 11:52:09.552648 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.552500    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2xwtg"]
Sep 11 11:52:09.555204 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:52:09.555180    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca51005b_bb90_4eac_8bc7_c64bb5463c3f.slice/crio-114bf75d672e9d6e6f48b2d0f86749c08812790ae85944514b5e7eb24907f765 WatchSource:0}: Error finding container 114bf75d672e9d6e6f48b2d0f86749c08812790ae85944514b5e7eb24907f765: Status 404 returned error can't find the container with id 114bf75d672e9d6e6f48b2d0f86749c08812790ae85944514b5e7eb24907f765
Sep 11 11:52:09.895452 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.895380    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 11:52:09.895743 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.895719    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7be71309-2439-482d-9202-9effbfb8fce3" containerName="console"
Sep 11 11:52:09.895743 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.895742    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7be71309-2439-482d-9202-9effbfb8fce3" containerName="console"
Sep 11 11:52:09.895909 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.895810    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7be71309-2439-482d-9202-9effbfb8fce3" containerName="console"
Sep 11 11:52:09.899140 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.899122    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:09.901690 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.901574    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 11:52:09.901690 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.901602    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 11:52:09.901690 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.901642    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 11:52:09.901690 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.901658    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 11:52:09.902045 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.901730    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dwwnq\""
Sep 11 11:52:09.902045 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.901748    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 11:52:09.902045 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.902000    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 11:52:09.902045 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.902014    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 11:52:09.902045 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.902025    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 11:52:09.906967 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.906765    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 11:52:09.918178 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:09.918013    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 11:52:10.010782 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.010751    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2xwtg" event={"ID":"ca51005b-bb90-4eac-8bc7-c64bb5463c3f","Type":"ContainerStarted","Data":"114bf75d672e9d6e6f48b2d0f86749c08812790ae85944514b5e7eb24907f765"}
Sep 11 11:52:10.015149 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.015117    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aebf0f41-2fa7-431b-b486-752354be0f44-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.015237 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.015157    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-config-volume\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.015237 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.015179    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.015350 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.015255    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.015350 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.015311    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aebf0f41-2fa7-431b-b486-752354be0f44-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.015432 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.015368    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.015432 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.015384    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shxx6\" (UniqueName: \"kubernetes.io/projected/aebf0f41-2fa7-431b-b486-752354be0f44-kube-api-access-shxx6\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.015432 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.015401    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aebf0f41-2fa7-431b-b486-752354be0f44-config-out\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.015432 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.015418    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.015432 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.015433    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aebf0f41-2fa7-431b-b486-752354be0f44-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.015616 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.015454    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-web-config\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.015616 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.015470    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.015616 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.015486    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aebf0f41-2fa7-431b-b486-752354be0f44-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.116488 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.116450    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aebf0f41-2fa7-431b-b486-752354be0f44-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.116687 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.116503    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-config-volume\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.116830 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.116761    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.116830 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.116808    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.117054 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.116859    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aebf0f41-2fa7-431b-b486-752354be0f44-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.117054 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.116928    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.117233 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.117187    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shxx6\" (UniqueName: \"kubernetes.io/projected/aebf0f41-2fa7-431b-b486-752354be0f44-kube-api-access-shxx6\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.117387 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.117369    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aebf0f41-2fa7-431b-b486-752354be0f44-config-out\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.117508 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.117492    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.117636 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.117620    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aebf0f41-2fa7-431b-b486-752354be0f44-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.117740 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.117549    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aebf0f41-2fa7-431b-b486-752354be0f44-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.117847 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.117832    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-web-config\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.117951 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.117935    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.118052 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.118039    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aebf0f41-2fa7-431b-b486-752354be0f44-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.118364 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.118336    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aebf0f41-2fa7-431b-b486-752354be0f44-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.118709 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.117207    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aebf0f41-2fa7-431b-b486-752354be0f44-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.119861 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.119739    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aebf0f41-2fa7-431b-b486-752354be0f44-config-out\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.119861 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.119785    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.120193 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.120166    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-config-volume\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.120351 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.120326    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.120902 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.120879    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.121271 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.121246    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aebf0f41-2fa7-431b-b486-752354be0f44-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.121616 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.121577    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.121850 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.121829    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-web-config\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.122247 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.122228    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aebf0f41-2fa7-431b-b486-752354be0f44-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.126807 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.126779    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shxx6\" (UniqueName: \"kubernetes.io/projected/aebf0f41-2fa7-431b-b486-752354be0f44-kube-api-access-shxx6\") pod \"alertmanager-main-0\" (UID: \"aebf0f41-2fa7-431b-b486-752354be0f44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.209027 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.208942    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 11:52:10.496422 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:10.496395    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 11:52:10.500349 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:52:10.500322    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaebf0f41_2fa7_431b_b486_752354be0f44.slice/crio-20fc89f2117f678aabb5859c0a2a614720941aee32da8c2cf47d09a0177e2b66 WatchSource:0}: Error finding container 20fc89f2117f678aabb5859c0a2a614720941aee32da8c2cf47d09a0177e2b66: Status 404 returned error can't find the container with id 20fc89f2117f678aabb5859c0a2a614720941aee32da8c2cf47d09a0177e2b66
Sep 11 11:52:11.016529 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:11.016500    2569 generic.go:356] "Generic (PLEG): container finished" podID="aebf0f41-2fa7-431b-b486-752354be0f44" containerID="64abeecc1cc956c8eb5a5257c5dbf16c964fc38ae5276f46cd6fdd445bc1f50b" exitCode=0
Sep 11 11:52:11.016944 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:11.016609    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aebf0f41-2fa7-431b-b486-752354be0f44","Type":"ContainerDied","Data":"64abeecc1cc956c8eb5a5257c5dbf16c964fc38ae5276f46cd6fdd445bc1f50b"}
Sep 11 11:52:11.016944 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:11.016690    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aebf0f41-2fa7-431b-b486-752354be0f44","Type":"ContainerStarted","Data":"20fc89f2117f678aabb5859c0a2a614720941aee32da8c2cf47d09a0177e2b66"}
Sep 11 11:52:11.018495 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:11.018473    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2xwtg" event={"ID":"ca51005b-bb90-4eac-8bc7-c64bb5463c3f","Type":"ContainerStarted","Data":"99f3ac1f65461574f8d0db02fd1764d247fbd7de8f95b4c704c90d9b32be1c9f"}
Sep 11 11:52:11.018574 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:11.018562    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2xwtg" event={"ID":"ca51005b-bb90-4eac-8bc7-c64bb5463c3f","Type":"ContainerStarted","Data":"fc758e3b75b6abb5bc0ceb6af4b287ba3858c276c8cef13053c25f6fa51aebad"}
Sep 11 11:52:11.059629 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:11.059574    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2xwtg" podStartSLOduration=252.094996887 podStartE2EDuration="4m13.059561596s" totalImagesPullingTime="964.564709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:52:11.058647199 +0000 UTC m=+253.480515605" watchObservedRunningTime="2026-09-11 11:52:11.059561596 +0000 UTC m=+253.481430003"
Sep 11 11:52:13.029915 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.029880    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aebf0f41-2fa7-431b-b486-752354be0f44","Type":"ContainerStarted","Data":"8f52c5d208d8ba349f298a6937cea50740d7bb0bf5af02438f4aecb3f421d654"}
Sep 11 11:52:13.030270 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.029971    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aebf0f41-2fa7-431b-b486-752354be0f44","Type":"ContainerStarted","Data":"2899af8c93d72988c342be1d0bf32b44654334331e7b40545987e6a56a89ddda"}
Sep 11 11:52:13.030270 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.029996    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aebf0f41-2fa7-431b-b486-752354be0f44","Type":"ContainerStarted","Data":"ab0892294f8c8bb00162e2eb7400fae1df5b738a761b6d963803b701909fb7ee"}
Sep 11 11:52:13.030270 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.030038    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aebf0f41-2fa7-431b-b486-752354be0f44","Type":"ContainerStarted","Data":"ebc888af9d1b1cf8d90c8e274fd453ae1ba397dc8c3c2c47c358673eaf00987a"}
Sep 11 11:52:13.030270 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.030062    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aebf0f41-2fa7-431b-b486-752354be0f44","Type":"ContainerStarted","Data":"1225ec6768f455f7450e5aec953f6db821d873c83fc5e4e9a8fdbf985ed51824"}
Sep 11 11:52:13.254692 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.254660    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-98b75cfb6-jjbrg"]
Sep 11 11:52:13.256860 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.256841    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.271386 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.271363    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-98b75cfb6-jjbrg"]
Sep 11 11:52:13.344633 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.344557    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-trusted-ca-bundle\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.344633 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.344612    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkn48\" (UniqueName: \"kubernetes.io/projected/4197d176-a6c4-490f-8353-0f37403bf815-kube-api-access-lkn48\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.344774 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.344649    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-oauth-serving-cert\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.344774 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.344679    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-console-config\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.344774 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.344751    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4197d176-a6c4-490f-8353-0f37403bf815-console-oauth-config\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.344891 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.344774    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-service-ca\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.344891 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.344797    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4197d176-a6c4-490f-8353-0f37403bf815-console-serving-cert\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.445671 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.445630    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4197d176-a6c4-490f-8353-0f37403bf815-console-serving-cert\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.445860 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.445716    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-trusted-ca-bundle\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.445860 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.445734    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkn48\" (UniqueName: \"kubernetes.io/projected/4197d176-a6c4-490f-8353-0f37403bf815-kube-api-access-lkn48\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.445860 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.445753    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-oauth-serving-cert\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.445860 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.445773    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-console-config\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.445860 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.445806    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4197d176-a6c4-490f-8353-0f37403bf815-console-oauth-config\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.446104 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.445896    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-service-ca\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.446629 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.446580    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-console-config\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.446737 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.446630    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-oauth-serving-cert\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.446737 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.446694    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-trusted-ca-bundle\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.446737 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.446695    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-service-ca\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.448258 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.448235    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4197d176-a6c4-490f-8353-0f37403bf815-console-serving-cert\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.448335 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.448266    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4197d176-a6c4-490f-8353-0f37403bf815-console-oauth-config\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.453818 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.453802    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkn48\" (UniqueName: \"kubernetes.io/projected/4197d176-a6c4-490f-8353-0f37403bf815-kube-api-access-lkn48\") pod \"console-98b75cfb6-jjbrg\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") " pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.565134 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.565093    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:13.834292 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:13.834243    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-98b75cfb6-jjbrg"]
Sep 11 11:52:13.836894 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:52:13.836871    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4197d176_a6c4_490f_8353_0f37403bf815.slice/crio-91f9e6b98db8f13bc7f917be88c49f981e1c59d89ce4bcc2553806d2e421885d WatchSource:0}: Error finding container 91f9e6b98db8f13bc7f917be88c49f981e1c59d89ce4bcc2553806d2e421885d: Status 404 returned error can't find the container with id 91f9e6b98db8f13bc7f917be88c49f981e1c59d89ce4bcc2553806d2e421885d
Sep 11 11:52:14.036129 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:14.036096    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-98b75cfb6-jjbrg" event={"ID":"4197d176-a6c4-490f-8353-0f37403bf815","Type":"ContainerStarted","Data":"96d6ab41bdea57ca92362b23f8c378020f3bb608cdb01b1eb836e49820c13a52"}
Sep 11 11:52:14.036578 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:14.036153    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-98b75cfb6-jjbrg" event={"ID":"4197d176-a6c4-490f-8353-0f37403bf815","Type":"ContainerStarted","Data":"91f9e6b98db8f13bc7f917be88c49f981e1c59d89ce4bcc2553806d2e421885d"}
Sep 11 11:52:14.056893 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:14.056836    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-98b75cfb6-jjbrg" podStartSLOduration=1.056815639 podStartE2EDuration="1.056815639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:52:14.054735443 +0000 UTC m=+256.476603839" watchObservedRunningTime="2026-09-11 11:52:14.056815639 +0000 UTC m=+256.478684227"
Sep 11 11:52:15.043750 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:15.043712    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aebf0f41-2fa7-431b-b486-752354be0f44","Type":"ContainerStarted","Data":"86268cb95175c71e593afb27a6a0eca8e8e6a3d4aa5a5ef548c8a7755dad49ab"}
Sep 11 11:52:15.087146 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:15.087085    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.278711383 podStartE2EDuration="6.087064538s" totalImagesPullingTime="2.788985642s" totalInitContainerRuntime="19.367513ms" isStatefulPod=false podCreationTimestamp="2026-09-11 11:52:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:52:15.086330697 +0000 UTC m=+257.508199088" watchObservedRunningTime="2026-09-11 11:52:15.087064538 +0000 UTC m=+257.508932919"
Sep 11 11:52:23.566228 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:23.566131    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:23.566228 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:23.566193    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:23.571142 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:23.571122    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:24.083279 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:24.083251    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:52:24.131818 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:24.131794    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-799859f498-kzw4d"]
Sep 11 11:52:49.150983 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.150919    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-799859f498-kzw4d" podUID="712bcd5d-48fa-4b5d-98c2-f78b60a788fb" containerName="console" containerID="cri-o://a9a7e096c73438e0058f27994ea01c77503f59af3522c4bc632fd9c73d343957" gracePeriod=15
Sep 11 11:52:49.380242 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.380216    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:52:49.510312 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.510286    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzb2q\" (UniqueName: \"kubernetes.io/projected/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-kube-api-access-vzb2q\") pod \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") "
Sep 11 11:52:49.510428 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.510321    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-oauth-config\") pod \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") "
Sep 11 11:52:49.510428 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.510340    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-serving-cert\") pod \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") "
Sep 11 11:52:49.510428 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.510375    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-oauth-serving-cert\") pod \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") "
Sep 11 11:52:49.510428 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.510403    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-trusted-ca-bundle\") pod \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") "
Sep 11 11:52:49.510577 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.510427    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-service-ca\") pod \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") "
Sep 11 11:52:49.510577 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.510500    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-config\") pod \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\" (UID: \"712bcd5d-48fa-4b5d-98c2-f78b60a788fb\") "
Sep 11 11:52:49.510790 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.510757    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "712bcd5d-48fa-4b5d-98c2-f78b60a788fb" (UID: "712bcd5d-48fa-4b5d-98c2-f78b60a788fb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 11:52:49.510790 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.510775    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "712bcd5d-48fa-4b5d-98c2-f78b60a788fb" (UID: "712bcd5d-48fa-4b5d-98c2-f78b60a788fb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 11:52:49.510976 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.510891    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-service-ca" (OuterVolumeSpecName: "service-ca") pod "712bcd5d-48fa-4b5d-98c2-f78b60a788fb" (UID: "712bcd5d-48fa-4b5d-98c2-f78b60a788fb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 11:52:49.510976 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.510963    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-config" (OuterVolumeSpecName: "console-config") pod "712bcd5d-48fa-4b5d-98c2-f78b60a788fb" (UID: "712bcd5d-48fa-4b5d-98c2-f78b60a788fb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 11:52:49.512374 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.512350    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "712bcd5d-48fa-4b5d-98c2-f78b60a788fb" (UID: "712bcd5d-48fa-4b5d-98c2-f78b60a788fb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 11:52:49.512476 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.512458    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-kube-api-access-vzb2q" (OuterVolumeSpecName: "kube-api-access-vzb2q") pod "712bcd5d-48fa-4b5d-98c2-f78b60a788fb" (UID: "712bcd5d-48fa-4b5d-98c2-f78b60a788fb"). InnerVolumeSpecName "kube-api-access-vzb2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:52:49.512535 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.512521    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "712bcd5d-48fa-4b5d-98c2-f78b60a788fb" (UID: "712bcd5d-48fa-4b5d-98c2-f78b60a788fb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 11:52:49.611500 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.611477    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-oauth-serving-cert\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:52:49.611500 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.611497    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-trusted-ca-bundle\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:52:49.611640 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.611507    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:52:49.611640 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.611515    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-config\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:52:49.611640 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.611523    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzb2q\" (UniqueName: \"kubernetes.io/projected/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-kube-api-access-vzb2q\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:52:49.611640 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.611532    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-oauth-config\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:52:49.611640 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:49.611540    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/712bcd5d-48fa-4b5d-98c2-f78b60a788fb-console-serving-cert\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:52:50.188371 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:50.188339    2569 generic.go:356] "Generic (PLEG): container finished" podID="712bcd5d-48fa-4b5d-98c2-f78b60a788fb" containerID="a9a7e096c73438e0058f27994ea01c77503f59af3522c4bc632fd9c73d343957" exitCode=0
Sep 11 11:52:50.188750 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:50.188392    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799859f498-kzw4d"
Sep 11 11:52:50.188750 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:50.188418    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799859f498-kzw4d" event={"ID":"712bcd5d-48fa-4b5d-98c2-f78b60a788fb","Type":"ContainerDied","Data":"a9a7e096c73438e0058f27994ea01c77503f59af3522c4bc632fd9c73d343957"}
Sep 11 11:52:50.188750 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:50.188480    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799859f498-kzw4d" event={"ID":"712bcd5d-48fa-4b5d-98c2-f78b60a788fb","Type":"ContainerDied","Data":"7893f2ed53220073237d428ce2cd3b80ece460d811ba73d0c5272c294504fd95"}
Sep 11 11:52:50.188750 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:50.188501    2569 scope.go:117] "RemoveContainer" containerID="a9a7e096c73438e0058f27994ea01c77503f59af3522c4bc632fd9c73d343957"
Sep 11 11:52:50.196504 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:50.196489    2569 scope.go:117] "RemoveContainer" containerID="a9a7e096c73438e0058f27994ea01c77503f59af3522c4bc632fd9c73d343957"
Sep 11 11:52:50.196803 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:52:50.196786    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9a7e096c73438e0058f27994ea01c77503f59af3522c4bc632fd9c73d343957\": container with ID starting with a9a7e096c73438e0058f27994ea01c77503f59af3522c4bc632fd9c73d343957 not found: ID does not exist" containerID="a9a7e096c73438e0058f27994ea01c77503f59af3522c4bc632fd9c73d343957"
Sep 11 11:52:50.196852 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:50.196811    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9a7e096c73438e0058f27994ea01c77503f59af3522c4bc632fd9c73d343957"} err="failed to get container status \"a9a7e096c73438e0058f27994ea01c77503f59af3522c4bc632fd9c73d343957\": rpc error: code = NotFound desc = could not find container \"a9a7e096c73438e0058f27994ea01c77503f59af3522c4bc632fd9c73d343957\": container with ID starting with a9a7e096c73438e0058f27994ea01c77503f59af3522c4bc632fd9c73d343957 not found: ID does not exist"
Sep 11 11:52:50.209217 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:50.209197    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-799859f498-kzw4d"]
Sep 11 11:52:50.215672 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:50.215653    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-799859f498-kzw4d"]
Sep 11 11:52:52.068524 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:52.068491    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="712bcd5d-48fa-4b5d-98c2-f78b60a788fb" path="/var/lib/kubelet/pods/712bcd5d-48fa-4b5d-98c2-f78b60a788fb/volumes"
Sep 11 11:52:58.013078 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:58.013047    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 11:52:58.019659 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:58.019634    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 11:52:58.020095 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:52:58.020071    2569 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 11:53:42.866483 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.866455    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp"]
Sep 11 11:53:42.866939 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.866785    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="712bcd5d-48fa-4b5d-98c2-f78b60a788fb" containerName="console"
Sep 11 11:53:42.866939 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.866798    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="712bcd5d-48fa-4b5d-98c2-f78b60a788fb" containerName="console"
Sep 11 11:53:42.866939 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.866866    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="712bcd5d-48fa-4b5d-98c2-f78b60a788fb" containerName="console"
Sep 11 11:53:42.868728 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.868712    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp"
Sep 11 11:53:42.872099 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.872077    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bwfjs\""
Sep 11 11:53:42.872772 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.872752    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 11:53:42.872772 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.872764    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 11:53:42.887599 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.887561    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp"]
Sep 11 11:53:42.887783 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.887759    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39d944d8-8e24-48e9-91ff-a33619c12a1d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp\" (UID: \"39d944d8-8e24-48e9-91ff-a33619c12a1d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp"
Sep 11 11:53:42.887842 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.887805    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39d944d8-8e24-48e9-91ff-a33619c12a1d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp\" (UID: \"39d944d8-8e24-48e9-91ff-a33619c12a1d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp"
Sep 11 11:53:42.887888 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.887858    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9hvt\" (UniqueName: \"kubernetes.io/projected/39d944d8-8e24-48e9-91ff-a33619c12a1d-kube-api-access-b9hvt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp\" (UID: \"39d944d8-8e24-48e9-91ff-a33619c12a1d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp"
Sep 11 11:53:42.988397 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.988372    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9hvt\" (UniqueName: \"kubernetes.io/projected/39d944d8-8e24-48e9-91ff-a33619c12a1d-kube-api-access-b9hvt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp\" (UID: \"39d944d8-8e24-48e9-91ff-a33619c12a1d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp"
Sep 11 11:53:42.988506 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.988415    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39d944d8-8e24-48e9-91ff-a33619c12a1d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp\" (UID: \"39d944d8-8e24-48e9-91ff-a33619c12a1d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp"
Sep 11 11:53:42.988506 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.988465    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39d944d8-8e24-48e9-91ff-a33619c12a1d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp\" (UID: \"39d944d8-8e24-48e9-91ff-a33619c12a1d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp"
Sep 11 11:53:42.988801 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.988784    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39d944d8-8e24-48e9-91ff-a33619c12a1d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp\" (UID: \"39d944d8-8e24-48e9-91ff-a33619c12a1d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp"
Sep 11 11:53:42.988846 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.988823    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39d944d8-8e24-48e9-91ff-a33619c12a1d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp\" (UID: \"39d944d8-8e24-48e9-91ff-a33619c12a1d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp"
Sep 11 11:53:42.999966 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:42.999941    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9hvt\" (UniqueName: \"kubernetes.io/projected/39d944d8-8e24-48e9-91ff-a33619c12a1d-kube-api-access-b9hvt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp\" (UID: \"39d944d8-8e24-48e9-91ff-a33619c12a1d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp"
Sep 11 11:53:43.178450 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:43.178398    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp"
Sep 11 11:53:43.472993 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:53:43.472949    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d944d8_8e24_48e9_91ff_a33619c12a1d.slice/crio-3e06c18b5744fb1b30e9c28157e5c72303d0992778cc2fad620ab03e5fa42533 WatchSource:0}: Error finding container 3e06c18b5744fb1b30e9c28157e5c72303d0992778cc2fad620ab03e5fa42533: Status 404 returned error can't find the container with id 3e06c18b5744fb1b30e9c28157e5c72303d0992778cc2fad620ab03e5fa42533
Sep 11 11:53:43.473398 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:43.473378    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp"]
Sep 11 11:53:43.474905 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:43.474889    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 11:53:44.396747 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:44.396713    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp" event={"ID":"39d944d8-8e24-48e9-91ff-a33619c12a1d","Type":"ContainerStarted","Data":"3e06c18b5744fb1b30e9c28157e5c72303d0992778cc2fad620ab03e5fa42533"}
Sep 11 11:53:49.420974 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:49.420909    2569 generic.go:356] "Generic (PLEG): container finished" podID="39d944d8-8e24-48e9-91ff-a33619c12a1d" containerID="b1dc3c340b54ed18a990f207a7bf4be92b063cc521fc64cc3721f8542a7a324a" exitCode=0
Sep 11 11:53:49.420974 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:49.420947    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp" event={"ID":"39d944d8-8e24-48e9-91ff-a33619c12a1d","Type":"ContainerDied","Data":"b1dc3c340b54ed18a990f207a7bf4be92b063cc521fc64cc3721f8542a7a324a"}
Sep 11 11:53:51.431862 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:51.431834    2569 generic.go:356] "Generic (PLEG): container finished" podID="39d944d8-8e24-48e9-91ff-a33619c12a1d" containerID="e12915eac928936f5f4a0e65434a72400a72a3dda2271b5698d3672b49479a1f" exitCode=0
Sep 11 11:53:51.432237 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:51.431891    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp" event={"ID":"39d944d8-8e24-48e9-91ff-a33619c12a1d","Type":"ContainerDied","Data":"e12915eac928936f5f4a0e65434a72400a72a3dda2271b5698d3672b49479a1f"}
Sep 11 11:53:59.470404 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:59.470368    2569 generic.go:356] "Generic (PLEG): container finished" podID="39d944d8-8e24-48e9-91ff-a33619c12a1d" containerID="131febdcecb38fc016843fc4a82e3e669dea10dbf2fa6a070c83262863ad7992" exitCode=0
Sep 11 11:53:59.470837 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:53:59.470448    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp" event={"ID":"39d944d8-8e24-48e9-91ff-a33619c12a1d","Type":"ContainerDied","Data":"131febdcecb38fc016843fc4a82e3e669dea10dbf2fa6a070c83262863ad7992"}
Sep 11 11:54:00.599248 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:00.599223    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp"
Sep 11 11:54:00.730041 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:00.729971    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9hvt\" (UniqueName: \"kubernetes.io/projected/39d944d8-8e24-48e9-91ff-a33619c12a1d-kube-api-access-b9hvt\") pod \"39d944d8-8e24-48e9-91ff-a33619c12a1d\" (UID: \"39d944d8-8e24-48e9-91ff-a33619c12a1d\") "
Sep 11 11:54:00.730041 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:00.730024    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39d944d8-8e24-48e9-91ff-a33619c12a1d-util\") pod \"39d944d8-8e24-48e9-91ff-a33619c12a1d\" (UID: \"39d944d8-8e24-48e9-91ff-a33619c12a1d\") "
Sep 11 11:54:00.730243 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:00.730107    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39d944d8-8e24-48e9-91ff-a33619c12a1d-bundle\") pod \"39d944d8-8e24-48e9-91ff-a33619c12a1d\" (UID: \"39d944d8-8e24-48e9-91ff-a33619c12a1d\") "
Sep 11 11:54:00.730790 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:00.730762    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d944d8-8e24-48e9-91ff-a33619c12a1d-bundle" (OuterVolumeSpecName: "bundle") pod "39d944d8-8e24-48e9-91ff-a33619c12a1d" (UID: "39d944d8-8e24-48e9-91ff-a33619c12a1d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:54:00.732164 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:00.732136    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d944d8-8e24-48e9-91ff-a33619c12a1d-kube-api-access-b9hvt" (OuterVolumeSpecName: "kube-api-access-b9hvt") pod "39d944d8-8e24-48e9-91ff-a33619c12a1d" (UID: "39d944d8-8e24-48e9-91ff-a33619c12a1d"). InnerVolumeSpecName "kube-api-access-b9hvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:54:00.734182 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:00.734155    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d944d8-8e24-48e9-91ff-a33619c12a1d-util" (OuterVolumeSpecName: "util") pod "39d944d8-8e24-48e9-91ff-a33619c12a1d" (UID: "39d944d8-8e24-48e9-91ff-a33619c12a1d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:54:00.831158 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:00.831135    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39d944d8-8e24-48e9-91ff-a33619c12a1d-util\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:54:00.831158 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:00.831155    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39d944d8-8e24-48e9-91ff-a33619c12a1d-bundle\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:54:00.831284 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:00.831166    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9hvt\" (UniqueName: \"kubernetes.io/projected/39d944d8-8e24-48e9-91ff-a33619c12a1d-kube-api-access-b9hvt\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:54:01.480668 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:01.480643    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp"
Sep 11 11:54:01.480818 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:01.480642    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pppdp" event={"ID":"39d944d8-8e24-48e9-91ff-a33619c12a1d","Type":"ContainerDied","Data":"3e06c18b5744fb1b30e9c28157e5c72303d0992778cc2fad620ab03e5fa42533"}
Sep 11 11:54:01.480818 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:01.480755    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e06c18b5744fb1b30e9c28157e5c72303d0992778cc2fad620ab03e5fa42533"
Sep 11 11:54:06.583038 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.583003    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vtw5c"]
Sep 11 11:54:06.583551 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.583359    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d944d8-8e24-48e9-91ff-a33619c12a1d" containerName="util"
Sep 11 11:54:06.583551 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.583375    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d944d8-8e24-48e9-91ff-a33619c12a1d" containerName="util"
Sep 11 11:54:06.583551 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.583385    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d944d8-8e24-48e9-91ff-a33619c12a1d" containerName="pull"
Sep 11 11:54:06.583551 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.583393    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d944d8-8e24-48e9-91ff-a33619c12a1d" containerName="pull"
Sep 11 11:54:06.583551 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.583400    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d944d8-8e24-48e9-91ff-a33619c12a1d" containerName="extract"
Sep 11 11:54:06.583551 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.583405    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d944d8-8e24-48e9-91ff-a33619c12a1d" containerName="extract"
Sep 11 11:54:06.583551 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.583465    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="39d944d8-8e24-48e9-91ff-a33619c12a1d" containerName="extract"
Sep 11 11:54:06.624378 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.624351    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vtw5c"]
Sep 11 11:54:06.624492 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.624430    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vtw5c"
Sep 11 11:54:06.627181 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.627149    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 11 11:54:06.627181 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.627164    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-ptc5p\""
Sep 11 11:54:06.627352 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.627200    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 11 11:54:06.675640 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.675613    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f971e93-c318-4419-b4a3-e3e6fe5226e2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vtw5c\" (UID: \"6f971e93-c318-4419-b4a3-e3e6fe5226e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vtw5c"
Sep 11 11:54:06.675747 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.675676    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbfcm\" (UniqueName: \"kubernetes.io/projected/6f971e93-c318-4419-b4a3-e3e6fe5226e2-kube-api-access-tbfcm\") pod \"cert-manager-operator-controller-manager-55c444bf86-vtw5c\" (UID: \"6f971e93-c318-4419-b4a3-e3e6fe5226e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vtw5c"
Sep 11 11:54:06.776871 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.776844    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbfcm\" (UniqueName: \"kubernetes.io/projected/6f971e93-c318-4419-b4a3-e3e6fe5226e2-kube-api-access-tbfcm\") pod \"cert-manager-operator-controller-manager-55c444bf86-vtw5c\" (UID: \"6f971e93-c318-4419-b4a3-e3e6fe5226e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vtw5c"
Sep 11 11:54:06.776963 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.776907    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f971e93-c318-4419-b4a3-e3e6fe5226e2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vtw5c\" (UID: \"6f971e93-c318-4419-b4a3-e3e6fe5226e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vtw5c"
Sep 11 11:54:06.777206 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.777191    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f971e93-c318-4419-b4a3-e3e6fe5226e2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vtw5c\" (UID: \"6f971e93-c318-4419-b4a3-e3e6fe5226e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vtw5c"
Sep 11 11:54:06.785871 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.785838    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbfcm\" (UniqueName: \"kubernetes.io/projected/6f971e93-c318-4419-b4a3-e3e6fe5226e2-kube-api-access-tbfcm\") pod \"cert-manager-operator-controller-manager-55c444bf86-vtw5c\" (UID: \"6f971e93-c318-4419-b4a3-e3e6fe5226e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vtw5c"
Sep 11 11:54:06.933124 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:06.933069    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vtw5c"
Sep 11 11:54:07.250964 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:07.250939    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vtw5c"]
Sep 11 11:54:07.254204 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:54:07.254180    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f971e93_c318_4419_b4a3_e3e6fe5226e2.slice/crio-00ab26381eb7dcf5ef7cc1ef6d02c694b2c629d5078a34d1ebb24001f57991bb WatchSource:0}: Error finding container 00ab26381eb7dcf5ef7cc1ef6d02c694b2c629d5078a34d1ebb24001f57991bb: Status 404 returned error can't find the container with id 00ab26381eb7dcf5ef7cc1ef6d02c694b2c629d5078a34d1ebb24001f57991bb
Sep 11 11:54:07.505962 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:07.505892    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vtw5c" event={"ID":"6f971e93-c318-4419-b4a3-e3e6fe5226e2","Type":"ContainerStarted","Data":"00ab26381eb7dcf5ef7cc1ef6d02c694b2c629d5078a34d1ebb24001f57991bb"}
Sep 11 11:54:10.521152 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:10.521118    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vtw5c" event={"ID":"6f971e93-c318-4419-b4a3-e3e6fe5226e2","Type":"ContainerStarted","Data":"dbdf21b2789e69740cab6e177fa3df853b5ff5c3004bd42a588b26bffdaaf90d"}
Sep 11 11:54:10.546311 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:10.546261    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vtw5c" podStartSLOduration=2.190117414 podStartE2EDuration="4.546246819s" totalImagesPullingTime="2.356129405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:54:10.543761643 +0000 UTC m=+372.965630023" watchObservedRunningTime="2026-09-11 11:54:10.546246819 +0000 UTC m=+372.968115199"
Sep 11 11:54:11.078440 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.078410    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rx7vq"]
Sep 11 11:54:11.100559 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.100535    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rx7vq"
Sep 11 11:54:11.102742 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.102713    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rx7vq"]
Sep 11 11:54:11.107311 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.107135    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 11:54:11.107311 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.107148    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 11:54:11.107456 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.107440    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-59zp5\""
Sep 11 11:54:11.212754 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.212734    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk"]
Sep 11 11:54:11.216471 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.216450    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15ba62c5-73a5-47ed-9d74-1feb44e53ad5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rx7vq\" (UID: \"15ba62c5-73a5-47ed-9d74-1feb44e53ad5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rx7vq"
Sep 11 11:54:11.216614 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.216511    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvg8f\" (UniqueName: \"kubernetes.io/projected/15ba62c5-73a5-47ed-9d74-1feb44e53ad5-kube-api-access-kvg8f\") pod \"cert-manager-webhook-6486ffd997-rx7vq\" (UID: \"15ba62c5-73a5-47ed-9d74-1feb44e53ad5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rx7vq"
Sep 11 11:54:11.236390 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.236370    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk"
Sep 11 11:54:11.238702 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.238684    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk"]
Sep 11 11:54:11.242296 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.242280    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 11:54:11.244178 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.243967    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 11:54:11.244178 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.244135    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bwfjs\""
Sep 11 11:54:11.317842 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.317820    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15ba62c5-73a5-47ed-9d74-1feb44e53ad5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rx7vq\" (UID: \"15ba62c5-73a5-47ed-9d74-1feb44e53ad5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rx7vq"
Sep 11 11:54:11.317933 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.317850    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvg8f\" (UniqueName: \"kubernetes.io/projected/15ba62c5-73a5-47ed-9d74-1feb44e53ad5-kube-api-access-kvg8f\") pod \"cert-manager-webhook-6486ffd997-rx7vq\" (UID: \"15ba62c5-73a5-47ed-9d74-1feb44e53ad5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rx7vq"
Sep 11 11:54:11.317933 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.317880    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk\" (UID: \"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk"
Sep 11 11:54:11.317933 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.317914    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk\" (UID: \"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk"
Sep 11 11:54:11.318055 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.317958    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2klt8\" (UniqueName: \"kubernetes.io/projected/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-kube-api-access-2klt8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk\" (UID: \"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk"
Sep 11 11:54:11.335719 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.335671    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvg8f\" (UniqueName: \"kubernetes.io/projected/15ba62c5-73a5-47ed-9d74-1feb44e53ad5-kube-api-access-kvg8f\") pod \"cert-manager-webhook-6486ffd997-rx7vq\" (UID: \"15ba62c5-73a5-47ed-9d74-1feb44e53ad5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rx7vq"
Sep 11 11:54:11.335719 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.335713    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15ba62c5-73a5-47ed-9d74-1feb44e53ad5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rx7vq\" (UID: \"15ba62c5-73a5-47ed-9d74-1feb44e53ad5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rx7vq"
Sep 11 11:54:11.418920 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.418899    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk\" (UID: \"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk"
Sep 11 11:54:11.419003 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.418931    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk\" (UID: \"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk"
Sep 11 11:54:11.419003 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.418961    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2klt8\" (UniqueName: \"kubernetes.io/projected/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-kube-api-access-2klt8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk\" (UID: \"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk"
Sep 11 11:54:11.419236 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.419220    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk\" (UID: \"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk"
Sep 11 11:54:11.419272 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.419251    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk\" (UID: \"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk"
Sep 11 11:54:11.420918 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.420894    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rx7vq"
Sep 11 11:54:11.444662 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.444635    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2klt8\" (UniqueName: \"kubernetes.io/projected/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-kube-api-access-2klt8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk\" (UID: \"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk"
Sep 11 11:54:11.546473 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.546451    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk"
Sep 11 11:54:11.736322 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.736291    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rx7vq"]
Sep 11 11:54:11.738618 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:54:11.738580    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15ba62c5_73a5_47ed_9d74_1feb44e53ad5.slice/crio-d7423bf29127a817d849588eba91e50dd3b3cf34dc474f99aaad350711f4b5b2 WatchSource:0}: Error finding container d7423bf29127a817d849588eba91e50dd3b3cf34dc474f99aaad350711f4b5b2: Status 404 returned error can't find the container with id d7423bf29127a817d849588eba91e50dd3b3cf34dc474f99aaad350711f4b5b2
Sep 11 11:54:11.857664 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:11.857642    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk"]
Sep 11 11:54:11.859077 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:54:11.859052    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0a6a05_66f9_4bc8_8b32_d330025ea6c4.slice/crio-b242549422d62a477ee6c1d3f697e9c98f6a633b922afcc9935dff71761542c7 WatchSource:0}: Error finding container b242549422d62a477ee6c1d3f697e9c98f6a633b922afcc9935dff71761542c7: Status 404 returned error can't find the container with id b242549422d62a477ee6c1d3f697e9c98f6a633b922afcc9935dff71761542c7
Sep 11 11:54:12.532980 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:12.532943    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rx7vq" event={"ID":"15ba62c5-73a5-47ed-9d74-1feb44e53ad5","Type":"ContainerStarted","Data":"d7423bf29127a817d849588eba91e50dd3b3cf34dc474f99aaad350711f4b5b2"}
Sep 11 11:54:12.534856 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:12.534799    2569 generic.go:356] "Generic (PLEG): container finished" podID="1a0a6a05-66f9-4bc8-8b32-d330025ea6c4" containerID="64037e1a3340618f378295f186d4581a4e5b647202ff67366974a1f8f96d37f0" exitCode=0
Sep 11 11:54:12.534856 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:12.534854    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk" event={"ID":"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4","Type":"ContainerDied","Data":"64037e1a3340618f378295f186d4581a4e5b647202ff67366974a1f8f96d37f0"}
Sep 11 11:54:12.535054 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:12.534892    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk" event={"ID":"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4","Type":"ContainerStarted","Data":"b242549422d62a477ee6c1d3f697e9c98f6a633b922afcc9935dff71761542c7"}
Sep 11 11:54:13.061730 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:13.061694    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-89j54"]
Sep 11 11:54:13.064375 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:13.064355    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89j54"
Sep 11 11:54:13.066770 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:13.066749    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-b5t7h\""
Sep 11 11:54:13.077474 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:13.077425    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-89j54"]
Sep 11 11:54:13.135486 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:13.135457    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6b6015d-8201-4556-9f99-195b18c7efaa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-89j54\" (UID: \"c6b6015d-8201-4556-9f99-195b18c7efaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89j54"
Sep 11 11:54:13.135637 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:13.135500    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdql6\" (UniqueName: \"kubernetes.io/projected/c6b6015d-8201-4556-9f99-195b18c7efaa-kube-api-access-zdql6\") pod \"cert-manager-cainjector-559cdd8c87-89j54\" (UID: \"c6b6015d-8201-4556-9f99-195b18c7efaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89j54"
Sep 11 11:54:13.236481 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:13.236449    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6b6015d-8201-4556-9f99-195b18c7efaa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-89j54\" (UID: \"c6b6015d-8201-4556-9f99-195b18c7efaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89j54"
Sep 11 11:54:13.236649 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:13.236488    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdql6\" (UniqueName: \"kubernetes.io/projected/c6b6015d-8201-4556-9f99-195b18c7efaa-kube-api-access-zdql6\") pod \"cert-manager-cainjector-559cdd8c87-89j54\" (UID: \"c6b6015d-8201-4556-9f99-195b18c7efaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89j54"
Sep 11 11:54:13.246170 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:13.246134    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6b6015d-8201-4556-9f99-195b18c7efaa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-89j54\" (UID: \"c6b6015d-8201-4556-9f99-195b18c7efaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89j54"
Sep 11 11:54:13.246387 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:13.246364    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdql6\" (UniqueName: \"kubernetes.io/projected/c6b6015d-8201-4556-9f99-195b18c7efaa-kube-api-access-zdql6\") pod \"cert-manager-cainjector-559cdd8c87-89j54\" (UID: \"c6b6015d-8201-4556-9f99-195b18c7efaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89j54"
Sep 11 11:54:13.378899 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:13.378820    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89j54"
Sep 11 11:54:15.272815 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:15.272795    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-89j54"]
Sep 11 11:54:15.274638 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:54:15.274613    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6b6015d_8201_4556_9f99_195b18c7efaa.slice/crio-25db7b37220401b2b4d52628b74d99defd265b87aeb3665744ffdbbcee0c5e3a WatchSource:0}: Error finding container 25db7b37220401b2b4d52628b74d99defd265b87aeb3665744ffdbbcee0c5e3a: Status 404 returned error can't find the container with id 25db7b37220401b2b4d52628b74d99defd265b87aeb3665744ffdbbcee0c5e3a
Sep 11 11:54:15.552962 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:15.552870    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rx7vq" event={"ID":"15ba62c5-73a5-47ed-9d74-1feb44e53ad5","Type":"ContainerStarted","Data":"318f14d451ead572544291a83df56bc5cfc31233b2c3bed27fca78ad7dfcba45"}
Sep 11 11:54:15.552962 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:15.552947    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rx7vq"
Sep 11 11:54:15.554868 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:15.554840    2569 generic.go:356] "Generic (PLEG): container finished" podID="1a0a6a05-66f9-4bc8-8b32-d330025ea6c4" containerID="46bf785c5a5eb8858886b9d7255f08f5daf5ae73289a161a7fa1152a71d94da8" exitCode=0
Sep 11 11:54:15.554995 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:15.554872    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk" event={"ID":"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4","Type":"ContainerDied","Data":"46bf785c5a5eb8858886b9d7255f08f5daf5ae73289a161a7fa1152a71d94da8"}
Sep 11 11:54:15.556751 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:15.556715    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89j54" event={"ID":"c6b6015d-8201-4556-9f99-195b18c7efaa","Type":"ContainerStarted","Data":"035da5e4ffdead553e1532136dbe2de50939d8c5fb1b77ba746540498e30c86f"}
Sep 11 11:54:15.556838 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:15.556755    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89j54" event={"ID":"c6b6015d-8201-4556-9f99-195b18c7efaa","Type":"ContainerStarted","Data":"25db7b37220401b2b4d52628b74d99defd265b87aeb3665744ffdbbcee0c5e3a"}
Sep 11 11:54:15.572474 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:15.572435    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-rx7vq" podStartSLOduration=1.297151195 podStartE2EDuration="4.572422747s" totalImagesPullingTime="3.275271552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:54:15.570262516 +0000 UTC m=+377.992130898" watchObservedRunningTime="2026-09-11 11:54:15.572422747 +0000 UTC m=+377.994291127"
Sep 11 11:54:15.587961 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:15.587926    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89j54" podStartSLOduration=2.587914406 podStartE2EDuration="2.587914406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:54:15.586295998 +0000 UTC m=+378.008164378" watchObservedRunningTime="2026-09-11 11:54:15.587914406 +0000 UTC m=+378.009782785"
Sep 11 11:54:16.563889 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:16.563851    2569 generic.go:356] "Generic (PLEG): container finished" podID="1a0a6a05-66f9-4bc8-8b32-d330025ea6c4" containerID="ab9a27b775a2f48ec6b2f1ff81956251748bf286ba984322281882609a5d227a" exitCode=0
Sep 11 11:54:16.564351 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:16.563932    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk" event={"ID":"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4","Type":"ContainerDied","Data":"ab9a27b775a2f48ec6b2f1ff81956251748bf286ba984322281882609a5d227a"}
Sep 11 11:54:17.694221 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:17.694199    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk"
Sep 11 11:54:17.776941 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:17.776912    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-bundle\") pod \"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4\" (UID: \"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4\") "
Sep 11 11:54:17.777085 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:17.776950    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-util\") pod \"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4\" (UID: \"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4\") "
Sep 11 11:54:17.777085 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:17.777022    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2klt8\" (UniqueName: \"kubernetes.io/projected/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-kube-api-access-2klt8\") pod \"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4\" (UID: \"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4\") "
Sep 11 11:54:17.777341 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:17.777319    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-bundle" (OuterVolumeSpecName: "bundle") pod "1a0a6a05-66f9-4bc8-8b32-d330025ea6c4" (UID: "1a0a6a05-66f9-4bc8-8b32-d330025ea6c4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:54:17.779031 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:17.779012    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-kube-api-access-2klt8" (OuterVolumeSpecName: "kube-api-access-2klt8") pod "1a0a6a05-66f9-4bc8-8b32-d330025ea6c4" (UID: "1a0a6a05-66f9-4bc8-8b32-d330025ea6c4"). InnerVolumeSpecName "kube-api-access-2klt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:54:17.781872 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:17.781853    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-util" (OuterVolumeSpecName: "util") pod "1a0a6a05-66f9-4bc8-8b32-d330025ea6c4" (UID: "1a0a6a05-66f9-4bc8-8b32-d330025ea6c4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:54:17.878552 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:17.878491    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2klt8\" (UniqueName: \"kubernetes.io/projected/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-kube-api-access-2klt8\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:54:17.878552 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:17.878522    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-bundle\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:54:17.878552 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:17.878531    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a0a6a05-66f9-4bc8-8b32-d330025ea6c4-util\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:54:18.576054 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:18.576020    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk" event={"ID":"1a0a6a05-66f9-4bc8-8b32-d330025ea6c4","Type":"ContainerDied","Data":"b242549422d62a477ee6c1d3f697e9c98f6a633b922afcc9935dff71761542c7"}
Sep 11 11:54:18.576219 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:18.576065    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b242549422d62a477ee6c1d3f697e9c98f6a633b922afcc9935dff71761542c7"
Sep 11 11:54:18.576219 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:18.576067    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fglvtk"
Sep 11 11:54:21.566185 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:21.566155    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rx7vq"
Sep 11 11:54:30.319448 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.319413    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss"]
Sep 11 11:54:30.319928 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.319745    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a0a6a05-66f9-4bc8-8b32-d330025ea6c4" containerName="util"
Sep 11 11:54:30.319928 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.319757    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a0a6a05-66f9-4bc8-8b32-d330025ea6c4" containerName="util"
Sep 11 11:54:30.319928 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.319775    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a0a6a05-66f9-4bc8-8b32-d330025ea6c4" containerName="extract"
Sep 11 11:54:30.319928 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.319781    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a0a6a05-66f9-4bc8-8b32-d330025ea6c4" containerName="extract"
Sep 11 11:54:30.319928 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.319796    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a0a6a05-66f9-4bc8-8b32-d330025ea6c4" containerName="pull"
Sep 11 11:54:30.319928 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.319802    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a0a6a05-66f9-4bc8-8b32-d330025ea6c4" containerName="pull"
Sep 11 11:54:30.319928 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.319858    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a0a6a05-66f9-4bc8-8b32-d330025ea6c4" containerName="extract"
Sep 11 11:54:30.322157 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.322141    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss"
Sep 11 11:54:30.325183 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.325143    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 11:54:30.325326 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.325223    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bwfjs\""
Sep 11 11:54:30.325326 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.325267    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 11:54:30.333910 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.333887    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss"]
Sep 11 11:54:30.470412 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.470385    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss\" (UID: \"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss"
Sep 11 11:54:30.470554 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.470417    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5f8b\" (UniqueName: \"kubernetes.io/projected/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-kube-api-access-v5f8b\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss\" (UID: \"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss"
Sep 11 11:54:30.470554 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.470454    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss\" (UID: \"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss"
Sep 11 11:54:30.571188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.571129    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss\" (UID: \"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss"
Sep 11 11:54:30.571188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.571159    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5f8b\" (UniqueName: \"kubernetes.io/projected/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-kube-api-access-v5f8b\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss\" (UID: \"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss"
Sep 11 11:54:30.571341 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.571194    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss\" (UID: \"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss"
Sep 11 11:54:30.571571 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.571555    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss\" (UID: \"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss"
Sep 11 11:54:30.571628 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.571551    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss\" (UID: \"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss"
Sep 11 11:54:30.592778 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.592758    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5f8b\" (UniqueName: \"kubernetes.io/projected/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-kube-api-access-v5f8b\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss\" (UID: \"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss"
Sep 11 11:54:30.631487 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.631466    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss"
Sep 11 11:54:30.921283 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:30.921088    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss"]
Sep 11 11:54:30.923886 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:54:30.923858    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5a542d_3f8d_4496_87a2_fe0cdc96d0c1.slice/crio-af6863d42479b483162e7565408480b39dacd0695b8431c9ccf63e8b6cc2b034 WatchSource:0}: Error finding container af6863d42479b483162e7565408480b39dacd0695b8431c9ccf63e8b6cc2b034: Status 404 returned error can't find the container with id af6863d42479b483162e7565408480b39dacd0695b8431c9ccf63e8b6cc2b034
Sep 11 11:54:31.637031 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:31.636998    2569 generic.go:356] "Generic (PLEG): container finished" podID="bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1" containerID="d7c87ec5d74ff50c7d97c3db68e7ab92dac5c02f3518c1943e2911f907f04295" exitCode=0
Sep 11 11:54:31.637392 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:31.637035    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss" event={"ID":"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1","Type":"ContainerDied","Data":"d7c87ec5d74ff50c7d97c3db68e7ab92dac5c02f3518c1943e2911f907f04295"}
Sep 11 11:54:31.637392 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:31.637081    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss" event={"ID":"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1","Type":"ContainerStarted","Data":"af6863d42479b483162e7565408480b39dacd0695b8431c9ccf63e8b6cc2b034"}
Sep 11 11:54:32.643939 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:32.643908    2569 generic.go:356] "Generic (PLEG): container finished" podID="bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1" containerID="14a48ca868770ac48d8e36d6dcc1f83d17ee8472411df23d6cda0dfeef025634" exitCode=0
Sep 11 11:54:32.644377 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:32.643970    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss" event={"ID":"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1","Type":"ContainerDied","Data":"14a48ca868770ac48d8e36d6dcc1f83d17ee8472411df23d6cda0dfeef025634"}
Sep 11 11:54:33.650377 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:33.650343    2569 generic.go:356] "Generic (PLEG): container finished" podID="bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1" containerID="10bc88b5f4be70f3464e7dec4483ec9d17851d9454927e683892d24efad6ceaf" exitCode=0
Sep 11 11:54:33.650791 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:33.650433    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss" event={"ID":"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1","Type":"ContainerDied","Data":"10bc88b5f4be70f3464e7dec4483ec9d17851d9454927e683892d24efad6ceaf"}
Sep 11 11:54:34.778883 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:34.778860    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss"
Sep 11 11:54:34.807395 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:34.807372    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-util\") pod \"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1\" (UID: \"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1\") "
Sep 11 11:54:34.807516 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:34.807422    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5f8b\" (UniqueName: \"kubernetes.io/projected/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-kube-api-access-v5f8b\") pod \"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1\" (UID: \"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1\") "
Sep 11 11:54:34.807516 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:34.807477    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-bundle\") pod \"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1\" (UID: \"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1\") "
Sep 11 11:54:34.808379 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:34.808348    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-bundle" (OuterVolumeSpecName: "bundle") pod "bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1" (UID: "bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:54:34.809638 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:34.809613    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-kube-api-access-v5f8b" (OuterVolumeSpecName: "kube-api-access-v5f8b") pod "bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1" (UID: "bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1"). InnerVolumeSpecName "kube-api-access-v5f8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:54:34.816250 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:34.816192    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-util" (OuterVolumeSpecName: "util") pod "bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1" (UID: "bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:54:34.909025 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:34.908964    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-bundle\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:54:34.909025 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:34.908995    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-util\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:54:34.909025 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:34.909004    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5f8b\" (UniqueName: \"kubernetes.io/projected/bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1-kube-api-access-v5f8b\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:54:35.661549 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:35.661520    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss"
Sep 11 11:54:35.661749 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:35.661549    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49sss" event={"ID":"bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1","Type":"ContainerDied","Data":"af6863d42479b483162e7565408480b39dacd0695b8431c9ccf63e8b6cc2b034"}
Sep 11 11:54:35.661749 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:35.661671    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af6863d42479b483162e7565408480b39dacd0695b8431c9ccf63e8b6cc2b034"
Sep 11 11:54:47.634008 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.633974    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8"]
Sep 11 11:54:47.634395 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.634284    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1" containerName="extract"
Sep 11 11:54:47.634395 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.634294    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1" containerName="extract"
Sep 11 11:54:47.634395 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.634305    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1" containerName="util"
Sep 11 11:54:47.634395 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.634312    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1" containerName="util"
Sep 11 11:54:47.634395 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.634328    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1" containerName="pull"
Sep 11 11:54:47.634395 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.634333    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1" containerName="pull"
Sep 11 11:54:47.634395 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.634387    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf5a542d-3f8d-4496-87a2-fe0cdc96d0c1" containerName="extract"
Sep 11 11:54:47.639877 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.639861    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8"
Sep 11 11:54:47.643278 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.643247    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 11:54:47.643392 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.643306    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bwfjs\""
Sep 11 11:54:47.644290 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.644271    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 11:54:47.650262 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.650241    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8"]
Sep 11 11:54:47.709467 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.709443    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l7zc\" (UniqueName: \"kubernetes.io/projected/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-kube-api-access-7l7zc\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8\" (UID: \"ea06b735-76a3-4669-bf4d-fcb57a0b05b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8"
Sep 11 11:54:47.709561 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.709496    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8\" (UID: \"ea06b735-76a3-4669-bf4d-fcb57a0b05b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8"
Sep 11 11:54:47.709647 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.709564    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8\" (UID: \"ea06b735-76a3-4669-bf4d-fcb57a0b05b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8"
Sep 11 11:54:47.810336 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.810311    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8\" (UID: \"ea06b735-76a3-4669-bf4d-fcb57a0b05b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8"
Sep 11 11:54:47.810461 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.810359    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l7zc\" (UniqueName: \"kubernetes.io/projected/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-kube-api-access-7l7zc\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8\" (UID: \"ea06b735-76a3-4669-bf4d-fcb57a0b05b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8"
Sep 11 11:54:47.810461 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.810402    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8\" (UID: \"ea06b735-76a3-4669-bf4d-fcb57a0b05b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8"
Sep 11 11:54:47.810691 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.810672    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8\" (UID: \"ea06b735-76a3-4669-bf4d-fcb57a0b05b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8"
Sep 11 11:54:47.810734 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.810701    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8\" (UID: \"ea06b735-76a3-4669-bf4d-fcb57a0b05b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8"
Sep 11 11:54:47.820506 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.820481    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l7zc\" (UniqueName: \"kubernetes.io/projected/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-kube-api-access-7l7zc\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8\" (UID: \"ea06b735-76a3-4669-bf4d-fcb57a0b05b6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8"
Sep 11 11:54:47.949017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.948960    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"]
Sep 11 11:54:47.949101 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.949027    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8"
Sep 11 11:54:47.952956 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.952942    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:47.971631 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.971609    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 11:54:47.972736 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.972706    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-47w2g\""
Sep 11 11:54:47.972736 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.972706    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 11:54:47.972911 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.972747    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 11:54:47.973229 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.973202    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 11:54:47.976630 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:47.976607    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"]
Sep 11 11:54:48.012860 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.012757    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/468a44cb-07a8-4625-bed8-ba64db6c29d7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fb8846894-krlxl\" (UID: \"468a44cb-07a8-4625-bed8-ba64db6c29d7\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:48.012860 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.012807    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/468a44cb-07a8-4625-bed8-ba64db6c29d7-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fb8846894-krlxl\" (UID: \"468a44cb-07a8-4625-bed8-ba64db6c29d7\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:48.013017 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.012956    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk5vh\" (UniqueName: \"kubernetes.io/projected/468a44cb-07a8-4625-bed8-ba64db6c29d7-kube-api-access-qk5vh\") pod \"opendatahub-operator-controller-manager-7fb8846894-krlxl\" (UID: \"468a44cb-07a8-4625-bed8-ba64db6c29d7\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:48.013067 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.013018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/468a44cb-07a8-4625-bed8-ba64db6c29d7-tmp\") pod \"opendatahub-operator-controller-manager-7fb8846894-krlxl\" (UID: \"468a44cb-07a8-4625-bed8-ba64db6c29d7\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:48.013113 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.013069    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/468a44cb-07a8-4625-bed8-ba64db6c29d7-manifests\") pod \"opendatahub-operator-controller-manager-7fb8846894-krlxl\" (UID: \"468a44cb-07a8-4625-bed8-ba64db6c29d7\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:48.113723 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.113692    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/468a44cb-07a8-4625-bed8-ba64db6c29d7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fb8846894-krlxl\" (UID: \"468a44cb-07a8-4625-bed8-ba64db6c29d7\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:48.113872 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.113726    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/468a44cb-07a8-4625-bed8-ba64db6c29d7-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fb8846894-krlxl\" (UID: \"468a44cb-07a8-4625-bed8-ba64db6c29d7\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:48.113872 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.113807    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk5vh\" (UniqueName: \"kubernetes.io/projected/468a44cb-07a8-4625-bed8-ba64db6c29d7-kube-api-access-qk5vh\") pod \"opendatahub-operator-controller-manager-7fb8846894-krlxl\" (UID: \"468a44cb-07a8-4625-bed8-ba64db6c29d7\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:48.113990 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.113871    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/468a44cb-07a8-4625-bed8-ba64db6c29d7-tmp\") pod \"opendatahub-operator-controller-manager-7fb8846894-krlxl\" (UID: \"468a44cb-07a8-4625-bed8-ba64db6c29d7\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:48.113990 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.113932    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/468a44cb-07a8-4625-bed8-ba64db6c29d7-manifests\") pod \"opendatahub-operator-controller-manager-7fb8846894-krlxl\" (UID: \"468a44cb-07a8-4625-bed8-ba64db6c29d7\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:48.114410 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.114382    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/468a44cb-07a8-4625-bed8-ba64db6c29d7-manifests\") pod \"opendatahub-operator-controller-manager-7fb8846894-krlxl\" (UID: \"468a44cb-07a8-4625-bed8-ba64db6c29d7\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:48.115964 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.115941    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/468a44cb-07a8-4625-bed8-ba64db6c29d7-tmp\") pod \"opendatahub-operator-controller-manager-7fb8846894-krlxl\" (UID: \"468a44cb-07a8-4625-bed8-ba64db6c29d7\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:48.116299 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.116281    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/468a44cb-07a8-4625-bed8-ba64db6c29d7-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fb8846894-krlxl\" (UID: \"468a44cb-07a8-4625-bed8-ba64db6c29d7\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:48.116352 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.116334    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/468a44cb-07a8-4625-bed8-ba64db6c29d7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fb8846894-krlxl\" (UID: \"468a44cb-07a8-4625-bed8-ba64db6c29d7\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:48.126619 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.126598    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk5vh\" (UniqueName: \"kubernetes.io/projected/468a44cb-07a8-4625-bed8-ba64db6c29d7-kube-api-access-qk5vh\") pod \"opendatahub-operator-controller-manager-7fb8846894-krlxl\" (UID: \"468a44cb-07a8-4625-bed8-ba64db6c29d7\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:48.245915 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.245893    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8"]
Sep 11 11:54:48.247391 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:54:48.247368    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea06b735_76a3_4669_bf4d_fcb57a0b05b6.slice/crio-4d3b08bfdcd2fa1e6689617868c2491dd7647950a4e22486f61703cd1281d7e7 WatchSource:0}: Error finding container 4d3b08bfdcd2fa1e6689617868c2491dd7647950a4e22486f61703cd1281d7e7: Status 404 returned error can't find the container with id 4d3b08bfdcd2fa1e6689617868c2491dd7647950a4e22486f61703cd1281d7e7
Sep 11 11:54:48.278806 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.278786    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:48.569681 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.569661    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"]
Sep 11 11:54:48.571547 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:54:48.571517    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468a44cb_07a8_4625_bed8_ba64db6c29d7.slice/crio-ba8707591f21a9e67aac367b347e4f25a2d73830f7018707b8897ea7a8325138 WatchSource:0}: Error finding container ba8707591f21a9e67aac367b347e4f25a2d73830f7018707b8897ea7a8325138: Status 404 returned error can't find the container with id ba8707591f21a9e67aac367b347e4f25a2d73830f7018707b8897ea7a8325138
Sep 11 11:54:48.721208 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.721166    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl" event={"ID":"468a44cb-07a8-4625-bed8-ba64db6c29d7","Type":"ContainerStarted","Data":"ba8707591f21a9e67aac367b347e4f25a2d73830f7018707b8897ea7a8325138"}
Sep 11 11:54:48.722776 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.722751    2569 generic.go:356] "Generic (PLEG): container finished" podID="ea06b735-76a3-4669-bf4d-fcb57a0b05b6" containerID="9f996e56f76938b0f0b1f93a000996bf7780d242a714d241ac78342db90d644b" exitCode=0
Sep 11 11:54:48.722895 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.722796    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8" event={"ID":"ea06b735-76a3-4669-bf4d-fcb57a0b05b6","Type":"ContainerDied","Data":"9f996e56f76938b0f0b1f93a000996bf7780d242a714d241ac78342db90d644b"}
Sep 11 11:54:48.722895 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:48.722837    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8" event={"ID":"ea06b735-76a3-4669-bf4d-fcb57a0b05b6","Type":"ContainerStarted","Data":"4d3b08bfdcd2fa1e6689617868c2491dd7647950a4e22486f61703cd1281d7e7"}
Sep 11 11:54:50.743150 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:50.743113    2569 generic.go:356] "Generic (PLEG): container finished" podID="ea06b735-76a3-4669-bf4d-fcb57a0b05b6" containerID="a2d5d0fec22eb3bca156cb1a7780a3e68fbc2f5cfb3da11893517a1f77ee0347" exitCode=0
Sep 11 11:54:50.743577 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:50.743156    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8" event={"ID":"ea06b735-76a3-4669-bf4d-fcb57a0b05b6","Type":"ContainerDied","Data":"a2d5d0fec22eb3bca156cb1a7780a3e68fbc2f5cfb3da11893517a1f77ee0347"}
Sep 11 11:54:51.751204 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:51.751170    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl" event={"ID":"468a44cb-07a8-4625-bed8-ba64db6c29d7","Type":"ContainerStarted","Data":"1b743bf6cdca74f01f980a1582a1d12a0dfa8f543be1faa56895159f4175c3db"}
Sep 11 11:54:51.753325 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:51.753303    2569 generic.go:356] "Generic (PLEG): container finished" podID="ea06b735-76a3-4669-bf4d-fcb57a0b05b6" containerID="a95d92cd4328e513faa89a7e01ad4b498ace32e8e5c7a4c105d6bfe5814c9d4e" exitCode=0
Sep 11 11:54:51.753412 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:51.753377    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8" event={"ID":"ea06b735-76a3-4669-bf4d-fcb57a0b05b6","Type":"ContainerDied","Data":"a95d92cd4328e513faa89a7e01ad4b498ace32e8e5c7a4c105d6bfe5814c9d4e"}
Sep 11 11:54:52.879569 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:52.879546    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8"
Sep 11 11:54:52.955332 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:52.955304    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l7zc\" (UniqueName: \"kubernetes.io/projected/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-kube-api-access-7l7zc\") pod \"ea06b735-76a3-4669-bf4d-fcb57a0b05b6\" (UID: \"ea06b735-76a3-4669-bf4d-fcb57a0b05b6\") "
Sep 11 11:54:52.955464 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:52.955365    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-util\") pod \"ea06b735-76a3-4669-bf4d-fcb57a0b05b6\" (UID: \"ea06b735-76a3-4669-bf4d-fcb57a0b05b6\") "
Sep 11 11:54:52.955464 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:52.955427    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-bundle\") pod \"ea06b735-76a3-4669-bf4d-fcb57a0b05b6\" (UID: \"ea06b735-76a3-4669-bf4d-fcb57a0b05b6\") "
Sep 11 11:54:52.956145 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:52.956114    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-bundle" (OuterVolumeSpecName: "bundle") pod "ea06b735-76a3-4669-bf4d-fcb57a0b05b6" (UID: "ea06b735-76a3-4669-bf4d-fcb57a0b05b6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:54:52.957339 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:52.957315    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-kube-api-access-7l7zc" (OuterVolumeSpecName: "kube-api-access-7l7zc") pod "ea06b735-76a3-4669-bf4d-fcb57a0b05b6" (UID: "ea06b735-76a3-4669-bf4d-fcb57a0b05b6"). InnerVolumeSpecName "kube-api-access-7l7zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:54:52.960794 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:52.960776    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-util" (OuterVolumeSpecName: "util") pod "ea06b735-76a3-4669-bf4d-fcb57a0b05b6" (UID: "ea06b735-76a3-4669-bf4d-fcb57a0b05b6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:54:53.056911 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:53.056861    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-bundle\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:54:53.056911 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:53.056882    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l7zc\" (UniqueName: \"kubernetes.io/projected/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-kube-api-access-7l7zc\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:54:53.056911 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:53.056892    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea06b735-76a3-4669-bf4d-fcb57a0b05b6-util\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:54:53.767416 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:53.767387    2569 generic.go:356] "Generic (PLEG): container finished" podID="468a44cb-07a8-4625-bed8-ba64db6c29d7" containerID="1b743bf6cdca74f01f980a1582a1d12a0dfa8f543be1faa56895159f4175c3db" exitCode=0
Sep 11 11:54:53.767614 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:53.767467    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl" event={"ID":"468a44cb-07a8-4625-bed8-ba64db6c29d7","Type":"ContainerDied","Data":"1b743bf6cdca74f01f980a1582a1d12a0dfa8f543be1faa56895159f4175c3db"}
Sep 11 11:54:53.769452 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:53.769431    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8"
Sep 11 11:54:53.769452 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:53.769441    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dzhr8" event={"ID":"ea06b735-76a3-4669-bf4d-fcb57a0b05b6","Type":"ContainerDied","Data":"4d3b08bfdcd2fa1e6689617868c2491dd7647950a4e22486f61703cd1281d7e7"}
Sep 11 11:54:53.769611 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:53.769473    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d3b08bfdcd2fa1e6689617868c2491dd7647950a4e22486f61703cd1281d7e7"
Sep 11 11:54:54.776569 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:54.776535    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl" event={"ID":"468a44cb-07a8-4625-bed8-ba64db6c29d7","Type":"ContainerStarted","Data":"8f79777e18faccf81cc52c2040207974461db44f343fd6fdca0ae33083a09f4e"}
Sep 11 11:54:54.776964 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:54.776771    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:54:54.803290 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:54:54.803240    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl" podStartSLOduration=3.314354272 podStartE2EDuration="7.80322578s" totalImagesPullingTime="3.415134872s" totalInitContainerRuntime="1.073736636s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:54:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:54:54.801667122 +0000 UTC m=+417.223535520" watchObservedRunningTime="2026-09-11 11:54:54.80322578 +0000 UTC m=+417.225094159"
Sep 11 11:55:05.784742 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:05.784708    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7fb8846894-krlxl"
Sep 11 11:55:13.550751 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.550717    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-57dff8dc5-hjn5h"]
Sep 11 11:55:13.551118 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.551037    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea06b735-76a3-4669-bf4d-fcb57a0b05b6" containerName="pull"
Sep 11 11:55:13.551118 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.551048    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea06b735-76a3-4669-bf4d-fcb57a0b05b6" containerName="pull"
Sep 11 11:55:13.551118 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.551063    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea06b735-76a3-4669-bf4d-fcb57a0b05b6" containerName="extract"
Sep 11 11:55:13.551118 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.551069    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea06b735-76a3-4669-bf4d-fcb57a0b05b6" containerName="extract"
Sep 11 11:55:13.551118 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.551082    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea06b735-76a3-4669-bf4d-fcb57a0b05b6" containerName="util"
Sep 11 11:55:13.551118 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.551088    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea06b735-76a3-4669-bf4d-fcb57a0b05b6" containerName="util"
Sep 11 11:55:13.551309 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.551138    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea06b735-76a3-4669-bf4d-fcb57a0b05b6" containerName="extract"
Sep 11 11:55:13.560675 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.560654    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57dff8dc5-hjn5h"
Sep 11 11:55:13.564343 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.564317    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 11:55:13.564343 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.564329    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 11:55:13.564504 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.564380    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 11:55:13.564504 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.564319    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gqrk7\""
Sep 11 11:55:13.564504 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.564320    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 11:55:13.570794 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.570774    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57dff8dc5-hjn5h"]
Sep 11 11:55:13.722268 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.722244    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rrzs\" (UniqueName: \"kubernetes.io/projected/047ffb0a-9d8a-42c1-a8ff-83f840a4bf80-kube-api-access-2rrzs\") pod \"kube-auth-proxy-57dff8dc5-hjn5h\" (UID: \"047ffb0a-9d8a-42c1-a8ff-83f840a4bf80\") " pod="openshift-ingress/kube-auth-proxy-57dff8dc5-hjn5h"
Sep 11 11:55:13.722406 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.722282    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/047ffb0a-9d8a-42c1-a8ff-83f840a4bf80-tmp\") pod \"kube-auth-proxy-57dff8dc5-hjn5h\" (UID: \"047ffb0a-9d8a-42c1-a8ff-83f840a4bf80\") " pod="openshift-ingress/kube-auth-proxy-57dff8dc5-hjn5h"
Sep 11 11:55:13.722406 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.722301    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047ffb0a-9d8a-42c1-a8ff-83f840a4bf80-tls-certs\") pod \"kube-auth-proxy-57dff8dc5-hjn5h\" (UID: \"047ffb0a-9d8a-42c1-a8ff-83f840a4bf80\") " pod="openshift-ingress/kube-auth-proxy-57dff8dc5-hjn5h"
Sep 11 11:55:13.823072 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.822994    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rrzs\" (UniqueName: \"kubernetes.io/projected/047ffb0a-9d8a-42c1-a8ff-83f840a4bf80-kube-api-access-2rrzs\") pod \"kube-auth-proxy-57dff8dc5-hjn5h\" (UID: \"047ffb0a-9d8a-42c1-a8ff-83f840a4bf80\") " pod="openshift-ingress/kube-auth-proxy-57dff8dc5-hjn5h"
Sep 11 11:55:13.823072 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.823030    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/047ffb0a-9d8a-42c1-a8ff-83f840a4bf80-tmp\") pod \"kube-auth-proxy-57dff8dc5-hjn5h\" (UID: \"047ffb0a-9d8a-42c1-a8ff-83f840a4bf80\") " pod="openshift-ingress/kube-auth-proxy-57dff8dc5-hjn5h"
Sep 11 11:55:13.823072 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.823048    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047ffb0a-9d8a-42c1-a8ff-83f840a4bf80-tls-certs\") pod \"kube-auth-proxy-57dff8dc5-hjn5h\" (UID: \"047ffb0a-9d8a-42c1-a8ff-83f840a4bf80\") " pod="openshift-ingress/kube-auth-proxy-57dff8dc5-hjn5h"
Sep 11 11:55:13.825276 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.825250    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/047ffb0a-9d8a-42c1-a8ff-83f840a4bf80-tmp\") pod \"kube-auth-proxy-57dff8dc5-hjn5h\" (UID: \"047ffb0a-9d8a-42c1-a8ff-83f840a4bf80\") " pod="openshift-ingress/kube-auth-proxy-57dff8dc5-hjn5h"
Sep 11 11:55:13.825522 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.825503    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047ffb0a-9d8a-42c1-a8ff-83f840a4bf80-tls-certs\") pod \"kube-auth-proxy-57dff8dc5-hjn5h\" (UID: \"047ffb0a-9d8a-42c1-a8ff-83f840a4bf80\") " pod="openshift-ingress/kube-auth-proxy-57dff8dc5-hjn5h"
Sep 11 11:55:13.832064 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.832045    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rrzs\" (UniqueName: \"kubernetes.io/projected/047ffb0a-9d8a-42c1-a8ff-83f840a4bf80-kube-api-access-2rrzs\") pod \"kube-auth-proxy-57dff8dc5-hjn5h\" (UID: \"047ffb0a-9d8a-42c1-a8ff-83f840a4bf80\") " pod="openshift-ingress/kube-auth-proxy-57dff8dc5-hjn5h"
Sep 11 11:55:13.871188 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:13.871163    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57dff8dc5-hjn5h"
Sep 11 11:55:14.147936 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:14.147915    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57dff8dc5-hjn5h"]
Sep 11 11:55:14.150119 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:55:14.150087    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047ffb0a_9d8a_42c1_a8ff_83f840a4bf80.slice/crio-549b51ccd48660d126e3dcb9202ba56cf4a88204649acdadb067777de8130b26 WatchSource:0}: Error finding container 549b51ccd48660d126e3dcb9202ba56cf4a88204649acdadb067777de8130b26: Status 404 returned error can't find the container with id 549b51ccd48660d126e3dcb9202ba56cf4a88204649acdadb067777de8130b26
Sep 11 11:55:14.873631 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:14.873578    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57dff8dc5-hjn5h" event={"ID":"047ffb0a-9d8a-42c1-a8ff-83f840a4bf80","Type":"ContainerStarted","Data":"549b51ccd48660d126e3dcb9202ba56cf4a88204649acdadb067777de8130b26"}
Sep 11 11:55:15.077973 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.077940    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"]
Sep 11 11:55:15.087705 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.087677    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.091053 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.091031    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 11:55:15.091704 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.091679    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 11:55:15.091819 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.091709    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 11:55:15.092025 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.092007    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fjhvz\""
Sep 11 11:55:15.092142 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.092061    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 11:55:15.098501 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.098478    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"]
Sep 11 11:55:15.237114 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.237033    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.237114 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.237080    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.237415 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.237147    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.237415 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.237169    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh4l5\" (UniqueName: \"kubernetes.io/projected/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-kube-api-access-wh4l5\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.237415 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.237230    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.237415 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.237284    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.237415 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.237316    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.337933 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.337899    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.338086 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.337939    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.338086 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.337966    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.338086 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.337989    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4l5\" (UniqueName: \"kubernetes.io/projected/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-kube-api-access-wh4l5\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.338086 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.338046    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.338292 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.338089    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.338292 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.338112    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.339078 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.339028    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.340469 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.340427    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.340832 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.340787    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.340937 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.340840    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.340937 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.340913    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.349013 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.348989    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.349468 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.349448    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh4l5\" (UniqueName: \"kubernetes.io/projected/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-kube-api-access-wh4l5\") pod \"istiod-openshift-gateway-7b7995d5cd-f6jfs\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.403363 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.403336    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:15.695572 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:15.695540    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"]
Sep 11 11:55:16.872662 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:55:16.872628    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f1e2c4_4e69_49b7_9a08_d93ff05546bd.slice/crio-ab244adfb23e23db2e80845fc5547f754737f41b9924cd41354f13d6182ba586 WatchSource:0}: Error finding container ab244adfb23e23db2e80845fc5547f754737f41b9924cd41354f13d6182ba586: Status 404 returned error can't find the container with id ab244adfb23e23db2e80845fc5547f754737f41b9924cd41354f13d6182ba586
Sep 11 11:55:16.891780 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:16.891692    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs" event={"ID":"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd","Type":"ContainerStarted","Data":"ab244adfb23e23db2e80845fc5547f754737f41b9924cd41354f13d6182ba586"}
Sep 11 11:55:17.897761 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:17.897717    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57dff8dc5-hjn5h" event={"ID":"047ffb0a-9d8a-42c1-a8ff-83f840a4bf80","Type":"ContainerStarted","Data":"403780f82bc8f59395ce0b0c82a7f1509bdac165bf2db86fdb93c4ea88419429"}
Sep 11 11:55:17.917155 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:17.917103    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-57dff8dc5-hjn5h" podStartSLOduration=2.126645578 podStartE2EDuration="4.917086178s" totalImagesPullingTime="2.7904406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:55:17.91552724 +0000 UTC m=+440.337395620" watchObservedRunningTime="2026-09-11 11:55:17.917086178 +0000 UTC m=+440.338954559"
Sep 11 11:55:19.855574 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:19.855535    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 11:55:19.855856 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:19.855625    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 11:55:20.916300 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:20.916211    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs" event={"ID":"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd","Type":"ContainerStarted","Data":"5de2f44ffafa16fbe7c2207e2ebd63a60a41831b3691fc250a0cc651b3eeaade"}
Sep 11 11:55:20.916722 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:20.916376    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:20.918130 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:20.918105    2569 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-f6jfs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 11:55:20.918231 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:20.918149    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs" podUID="f4f1e2c4-4e69-49b7-9a08-d93ff05546bd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 11:55:20.959784 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:20.959742    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs" podStartSLOduration=2.985446504 podStartE2EDuration="5.959729715s" totalImagesPullingTime="2.974283211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:55:20.958675194 +0000 UTC m=+443.380543603" watchObservedRunningTime="2026-09-11 11:55:20.959729715 +0000 UTC m=+443.381598095"
Sep 11 11:55:21.921845 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:21.921819    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:26.374836 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.374804    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"]
Sep 11 11:55:26.378788 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.378765    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"
Sep 11 11:55:26.381516 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.381337    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 11:55:26.382350 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.382325    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 11:55:26.382469 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.382377    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 11:55:26.382469 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.382385    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 11:55:26.382469 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.382424    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 11:55:26.382640 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.382542    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2fddf\""
Sep 11 11:55:26.392306 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.392285    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"]
Sep 11 11:55:26.425702 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.425677    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5cq7\" (UniqueName: \"kubernetes.io/projected/288653f1-8896-4c1c-990a-ecac00f6f0e3-kube-api-access-x5cq7\") pod \"lws-controller-manager-866dd874d6-sgpmk\" (UID: \"288653f1-8896-4c1c-990a-ecac00f6f0e3\") " pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"
Sep 11 11:55:26.425809 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.425717    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/288653f1-8896-4c1c-990a-ecac00f6f0e3-manager-config\") pod \"lws-controller-manager-866dd874d6-sgpmk\" (UID: \"288653f1-8896-4c1c-990a-ecac00f6f0e3\") " pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"
Sep 11 11:55:26.425809 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.425774    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/288653f1-8896-4c1c-990a-ecac00f6f0e3-metrics-cert\") pod \"lws-controller-manager-866dd874d6-sgpmk\" (UID: \"288653f1-8896-4c1c-990a-ecac00f6f0e3\") " pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"
Sep 11 11:55:26.425809 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.425804    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/288653f1-8896-4c1c-990a-ecac00f6f0e3-cert\") pod \"lws-controller-manager-866dd874d6-sgpmk\" (UID: \"288653f1-8896-4c1c-990a-ecac00f6f0e3\") " pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"
Sep 11 11:55:26.526963 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.526933    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5cq7\" (UniqueName: \"kubernetes.io/projected/288653f1-8896-4c1c-990a-ecac00f6f0e3-kube-api-access-x5cq7\") pod \"lws-controller-manager-866dd874d6-sgpmk\" (UID: \"288653f1-8896-4c1c-990a-ecac00f6f0e3\") " pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"
Sep 11 11:55:26.527091 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.526982    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/288653f1-8896-4c1c-990a-ecac00f6f0e3-manager-config\") pod \"lws-controller-manager-866dd874d6-sgpmk\" (UID: \"288653f1-8896-4c1c-990a-ecac00f6f0e3\") " pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"
Sep 11 11:55:26.527091 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.527031    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/288653f1-8896-4c1c-990a-ecac00f6f0e3-metrics-cert\") pod \"lws-controller-manager-866dd874d6-sgpmk\" (UID: \"288653f1-8896-4c1c-990a-ecac00f6f0e3\") " pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"
Sep 11 11:55:26.527091 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.527061    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/288653f1-8896-4c1c-990a-ecac00f6f0e3-cert\") pod \"lws-controller-manager-866dd874d6-sgpmk\" (UID: \"288653f1-8896-4c1c-990a-ecac00f6f0e3\") " pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"
Sep 11 11:55:26.527630 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.527610    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/288653f1-8896-4c1c-990a-ecac00f6f0e3-manager-config\") pod \"lws-controller-manager-866dd874d6-sgpmk\" (UID: \"288653f1-8896-4c1c-990a-ecac00f6f0e3\") " pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"
Sep 11 11:55:26.529580 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.529561    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/288653f1-8896-4c1c-990a-ecac00f6f0e3-metrics-cert\") pod \"lws-controller-manager-866dd874d6-sgpmk\" (UID: \"288653f1-8896-4c1c-990a-ecac00f6f0e3\") " pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"
Sep 11 11:55:26.529675 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.529562    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/288653f1-8896-4c1c-990a-ecac00f6f0e3-cert\") pod \"lws-controller-manager-866dd874d6-sgpmk\" (UID: \"288653f1-8896-4c1c-990a-ecac00f6f0e3\") " pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"
Sep 11 11:55:26.543218 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.543195    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5cq7\" (UniqueName: \"kubernetes.io/projected/288653f1-8896-4c1c-990a-ecac00f6f0e3-kube-api-access-x5cq7\") pod \"lws-controller-manager-866dd874d6-sgpmk\" (UID: \"288653f1-8896-4c1c-990a-ecac00f6f0e3\") " pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"
Sep 11 11:55:26.688147 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.688058    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"
Sep 11 11:55:26.971144 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:26.971115    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"]
Sep 11 11:55:26.973070 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:55:26.973042    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288653f1_8896_4c1c_990a_ecac00f6f0e3.slice/crio-5eac2b8befb2ddb106b595faeba1cfc827471b6abd6e5b7ba98626c730ba0f4d WatchSource:0}: Error finding container 5eac2b8befb2ddb106b595faeba1cfc827471b6abd6e5b7ba98626c730ba0f4d: Status 404 returned error can't find the container with id 5eac2b8befb2ddb106b595faeba1cfc827471b6abd6e5b7ba98626c730ba0f4d
Sep 11 11:55:27.953359 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:27.953323    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk" event={"ID":"288653f1-8896-4c1c-990a-ecac00f6f0e3","Type":"ContainerStarted","Data":"5eac2b8befb2ddb106b595faeba1cfc827471b6abd6e5b7ba98626c730ba0f4d"}
Sep 11 11:55:28.960970 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:28.960890    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk" event={"ID":"288653f1-8896-4c1c-990a-ecac00f6f0e3","Type":"ContainerStarted","Data":"777962fa242bd2a3683b7ad1bf91e11a91a01ed6f23ab65c41fffbccddaa85ad"}
Sep 11 11:55:28.961368 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:28.961042    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"
Sep 11 11:55:29.017754 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:29.017709    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk" podStartSLOduration=1.463282243 podStartE2EDuration="3.017697729s" totalImagesPullingTime="1.554415486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:55:29.017606041 +0000 UTC m=+451.439474435" watchObservedRunningTime="2026-09-11 11:55:29.017697729 +0000 UTC m=+451.439566108"
Sep 11 11:55:33.469847 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.469811    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"]
Sep 11 11:55:33.470220 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.469994    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs" podUID="f4f1e2c4-4e69-49b7-9a08-d93ff05546bd" containerName="discovery" containerID="cri-o://5de2f44ffafa16fbe7c2207e2ebd63a60a41831b3691fc250a0cc651b3eeaade" gracePeriod=30
Sep 11 11:55:33.705209 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.705187    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:33.790950 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.790928    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh4l5\" (UniqueName: \"kubernetes.io/projected/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-kube-api-access-wh4l5\") pod \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") "
Sep 11 11:55:33.791070 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.790985    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-cacerts\") pod \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") "
Sep 11 11:55:33.791070 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.791031    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-local-certs\") pod \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") "
Sep 11 11:55:33.791146 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.791084    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-csr-ca-configmap\") pod \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") "
Sep 11 11:55:33.791146 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.791116    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-kubeconfig\") pod \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") "
Sep 11 11:55:33.791223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.791166    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-token\") pod \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") "
Sep 11 11:55:33.791276 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.791252    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-csr-dns-cert\") pod \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\" (UID: \"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd\") "
Sep 11 11:55:33.791549 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.791520    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "f4f1e2c4-4e69-49b7-9a08-d93ff05546bd" (UID: "f4f1e2c4-4e69-49b7-9a08-d93ff05546bd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 11:55:33.793517 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.793492    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-local-certs" (OuterVolumeSpecName: "local-certs") pod "f4f1e2c4-4e69-49b7-9a08-d93ff05546bd" (UID: "f4f1e2c4-4e69-49b7-9a08-d93ff05546bd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:55:33.793517 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.793491    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "f4f1e2c4-4e69-49b7-9a08-d93ff05546bd" (UID: "f4f1e2c4-4e69-49b7-9a08-d93ff05546bd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 11:55:33.793687 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.793625    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-kube-api-access-wh4l5" (OuterVolumeSpecName: "kube-api-access-wh4l5") pod "f4f1e2c4-4e69-49b7-9a08-d93ff05546bd" (UID: "f4f1e2c4-4e69-49b7-9a08-d93ff05546bd"). InnerVolumeSpecName "kube-api-access-wh4l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:55:33.793687 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.793658    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-token" (OuterVolumeSpecName: "istio-token") pod "f4f1e2c4-4e69-49b7-9a08-d93ff05546bd" (UID: "f4f1e2c4-4e69-49b7-9a08-d93ff05546bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:55:33.793792 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.793776    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-cacerts" (OuterVolumeSpecName: "cacerts") pod "f4f1e2c4-4e69-49b7-9a08-d93ff05546bd" (UID: "f4f1e2c4-4e69-49b7-9a08-d93ff05546bd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 11:55:33.793830 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.793797    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "f4f1e2c4-4e69-49b7-9a08-d93ff05546bd" (UID: "f4f1e2c4-4e69-49b7-9a08-d93ff05546bd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 11:55:33.891914 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.891892    2569 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-cacerts\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:55:33.891914 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.891912    2569 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-local-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:55:33.892034 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.891922    2569 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-csr-ca-configmap\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:55:33.892034 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.891931    2569 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-kubeconfig\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:55:33.892034 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.891939    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-token\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:55:33.892034 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.891947    2569 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-istio-csr-dns-cert\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:55:33.892034 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.891956    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh4l5\" (UniqueName: \"kubernetes.io/projected/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd-kube-api-access-wh4l5\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:55:33.985739 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.985709    2569 generic.go:356] "Generic (PLEG): container finished" podID="f4f1e2c4-4e69-49b7-9a08-d93ff05546bd" containerID="5de2f44ffafa16fbe7c2207e2ebd63a60a41831b3691fc250a0cc651b3eeaade" exitCode=0
Sep 11 11:55:33.985848 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.985754    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"
Sep 11 11:55:33.985848 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.985800    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs" event={"ID":"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd","Type":"ContainerDied","Data":"5de2f44ffafa16fbe7c2207e2ebd63a60a41831b3691fc250a0cc651b3eeaade"}
Sep 11 11:55:33.985961 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.985855    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs" event={"ID":"f4f1e2c4-4e69-49b7-9a08-d93ff05546bd","Type":"ContainerDied","Data":"ab244adfb23e23db2e80845fc5547f754737f41b9924cd41354f13d6182ba586"}
Sep 11 11:55:33.985961 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.985885    2569 scope.go:117] "RemoveContainer" containerID="5de2f44ffafa16fbe7c2207e2ebd63a60a41831b3691fc250a0cc651b3eeaade"
Sep 11 11:55:33.994648 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.994628    2569 scope.go:117] "RemoveContainer" containerID="5de2f44ffafa16fbe7c2207e2ebd63a60a41831b3691fc250a0cc651b3eeaade"
Sep 11 11:55:33.994909 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:55:33.994887    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de2f44ffafa16fbe7c2207e2ebd63a60a41831b3691fc250a0cc651b3eeaade\": container with ID starting with 5de2f44ffafa16fbe7c2207e2ebd63a60a41831b3691fc250a0cc651b3eeaade not found: ID does not exist" containerID="5de2f44ffafa16fbe7c2207e2ebd63a60a41831b3691fc250a0cc651b3eeaade"
Sep 11 11:55:33.994989 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:33.994916    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de2f44ffafa16fbe7c2207e2ebd63a60a41831b3691fc250a0cc651b3eeaade"} err="failed to get container status \"5de2f44ffafa16fbe7c2207e2ebd63a60a41831b3691fc250a0cc651b3eeaade\": rpc error: code = NotFound desc = could not find container \"5de2f44ffafa16fbe7c2207e2ebd63a60a41831b3691fc250a0cc651b3eeaade\": container with ID starting with 5de2f44ffafa16fbe7c2207e2ebd63a60a41831b3691fc250a0cc651b3eeaade not found: ID does not exist"
Sep 11 11:55:34.015912 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:34.015888    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"]
Sep 11 11:55:34.021350 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:34.021330    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6jfs"]
Sep 11 11:55:34.068951 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:34.068901    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4f1e2c4-4e69-49b7-9a08-d93ff05546bd" path="/var/lib/kubelet/pods/f4f1e2c4-4e69-49b7-9a08-d93ff05546bd/volumes"
Sep 11 11:55:39.967945 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:55:39.967914    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-866dd874d6-sgpmk"
Sep 11 11:56:24.486186 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.486148    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx"]
Sep 11 11:56:24.486581 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.486514    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4f1e2c4-4e69-49b7-9a08-d93ff05546bd" containerName="discovery"
Sep 11 11:56:24.486581 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.486525    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4f1e2c4-4e69-49b7-9a08-d93ff05546bd" containerName="discovery"
Sep 11 11:56:24.486581 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.486598    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4f1e2c4-4e69-49b7-9a08-d93ff05546bd" containerName="discovery"
Sep 11 11:56:24.489857 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.489841    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx"
Sep 11 11:56:24.492305 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.492282    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 11:56:24.492428 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.492346    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 11:56:24.493137 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.493121    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bwfjs\""
Sep 11 11:56:24.499715 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.499694    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx"]
Sep 11 11:56:24.580122 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.580097    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh"]
Sep 11 11:56:24.583646 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.583632    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh"
Sep 11 11:56:24.594510 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.594487    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh"]
Sep 11 11:56:24.597868 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.597848    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca75ca39-1cd9-43d9-980f-e4da6dd51780-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx\" (UID: \"ca75ca39-1cd9-43d9-980f-e4da6dd51780\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx"
Sep 11 11:56:24.597977 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.597886    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-244bd\" (UniqueName: \"kubernetes.io/projected/ca75ca39-1cd9-43d9-980f-e4da6dd51780-kube-api-access-244bd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx\" (UID: \"ca75ca39-1cd9-43d9-980f-e4da6dd51780\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx"
Sep 11 11:56:24.597977 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.597949    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca75ca39-1cd9-43d9-980f-e4da6dd51780-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx\" (UID: \"ca75ca39-1cd9-43d9-980f-e4da6dd51780\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx"
Sep 11 11:56:24.677707 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.677685    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n"]
Sep 11 11:56:24.681051 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.681036    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n"
Sep 11 11:56:24.691701 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.691680    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n"]
Sep 11 11:56:24.698913 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.698892    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca75ca39-1cd9-43d9-980f-e4da6dd51780-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx\" (UID: \"ca75ca39-1cd9-43d9-980f-e4da6dd51780\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx"
Sep 11 11:56:24.698995 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.698942    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9qpb\" (UniqueName: \"kubernetes.io/projected/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-kube-api-access-t9qpb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh\" (UID: \"76c483c7-01b3-4d1b-873a-f91e0ff20e6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh"
Sep 11 11:56:24.698995 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.698962    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh\" (UID: \"76c483c7-01b3-4d1b-873a-f91e0ff20e6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh"
Sep 11 11:56:24.699077 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.699060    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca75ca39-1cd9-43d9-980f-e4da6dd51780-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx\" (UID: \"ca75ca39-1cd9-43d9-980f-e4da6dd51780\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx"
Sep 11 11:56:24.699113 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.699102    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh\" (UID: \"76c483c7-01b3-4d1b-873a-f91e0ff20e6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh"
Sep 11 11:56:24.699155 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.699140    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-244bd\" (UniqueName: \"kubernetes.io/projected/ca75ca39-1cd9-43d9-980f-e4da6dd51780-kube-api-access-244bd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx\" (UID: \"ca75ca39-1cd9-43d9-980f-e4da6dd51780\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx"
Sep 11 11:56:24.699334 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.699314    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca75ca39-1cd9-43d9-980f-e4da6dd51780-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx\" (UID: \"ca75ca39-1cd9-43d9-980f-e4da6dd51780\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx"
Sep 11 11:56:24.699408 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.699390    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca75ca39-1cd9-43d9-980f-e4da6dd51780-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx\" (UID: \"ca75ca39-1cd9-43d9-980f-e4da6dd51780\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx"
Sep 11 11:56:24.708663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.708640    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-244bd\" (UniqueName: \"kubernetes.io/projected/ca75ca39-1cd9-43d9-980f-e4da6dd51780-kube-api-access-244bd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx\" (UID: \"ca75ca39-1cd9-43d9-980f-e4da6dd51780\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx"
Sep 11 11:56:24.784068 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.784044    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq"]
Sep 11 11:56:24.787746 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.787731    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq"
Sep 11 11:56:24.797406 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.797388    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq"]
Sep 11 11:56:24.800048 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.800031    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx"
Sep 11 11:56:24.800221 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.800205    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9qpb\" (UniqueName: \"kubernetes.io/projected/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-kube-api-access-t9qpb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh\" (UID: \"76c483c7-01b3-4d1b-873a-f91e0ff20e6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh"
Sep 11 11:56:24.800263 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.800232    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06249ccc-5c2a-401b-8e4b-6191104e0e2a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n\" (UID: \"06249ccc-5c2a-401b-8e4b-6191104e0e2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n"
Sep 11 11:56:24.800263 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.800250    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh\" (UID: \"76c483c7-01b3-4d1b-873a-f91e0ff20e6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh"
Sep 11 11:56:24.800353 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.800274    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgvgr\" (UniqueName: \"kubernetes.io/projected/06249ccc-5c2a-401b-8e4b-6191104e0e2a-kube-api-access-mgvgr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n\" (UID: \"06249ccc-5c2a-401b-8e4b-6191104e0e2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n"
Sep 11 11:56:24.800353 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.800295    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh\" (UID: \"76c483c7-01b3-4d1b-873a-f91e0ff20e6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh"
Sep 11 11:56:24.800353 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.800330    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06249ccc-5c2a-401b-8e4b-6191104e0e2a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n\" (UID: \"06249ccc-5c2a-401b-8e4b-6191104e0e2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n"
Sep 11 11:56:24.800664 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.800647    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh\" (UID: \"76c483c7-01b3-4d1b-873a-f91e0ff20e6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh"
Sep 11 11:56:24.800778 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.800760    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh\" (UID: \"76c483c7-01b3-4d1b-873a-f91e0ff20e6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh"
Sep 11 11:56:24.809486 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.809467    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9qpb\" (UniqueName: \"kubernetes.io/projected/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-kube-api-access-t9qpb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh\" (UID: \"76c483c7-01b3-4d1b-873a-f91e0ff20e6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh"
Sep 11 11:56:24.891689 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.891654    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh"
Sep 11 11:56:24.900983 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.900953    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzh2h\" (UniqueName: \"kubernetes.io/projected/eef24a40-2739-487d-b111-66c0312a121c-kube-api-access-vzh2h\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq\" (UID: \"eef24a40-2739-487d-b111-66c0312a121c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq"
Sep 11 11:56:24.901094 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.901009    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eef24a40-2739-487d-b111-66c0312a121c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq\" (UID: \"eef24a40-2739-487d-b111-66c0312a121c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq"
Sep 11 11:56:24.901094 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.901067    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06249ccc-5c2a-401b-8e4b-6191104e0e2a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n\" (UID: \"06249ccc-5c2a-401b-8e4b-6191104e0e2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n"
Sep 11 11:56:24.901094 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.901083    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eef24a40-2739-487d-b111-66c0312a121c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq\" (UID: \"eef24a40-2739-487d-b111-66c0312a121c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq"
Sep 11 11:56:24.901250 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.901141    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgvgr\" (UniqueName: \"kubernetes.io/projected/06249ccc-5c2a-401b-8e4b-6191104e0e2a-kube-api-access-mgvgr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n\" (UID: \"06249ccc-5c2a-401b-8e4b-6191104e0e2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n"
Sep 11 11:56:24.901250 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.901188    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06249ccc-5c2a-401b-8e4b-6191104e0e2a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n\" (UID: \"06249ccc-5c2a-401b-8e4b-6191104e0e2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n"
Sep 11 11:56:24.901418 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.901398    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06249ccc-5c2a-401b-8e4b-6191104e0e2a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n\" (UID: \"06249ccc-5c2a-401b-8e4b-6191104e0e2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n"
Sep 11 11:56:24.901476 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.901435    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06249ccc-5c2a-401b-8e4b-6191104e0e2a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n\" (UID: \"06249ccc-5c2a-401b-8e4b-6191104e0e2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n"
Sep 11 11:56:24.912028 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.912007    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgvgr\" (UniqueName: \"kubernetes.io/projected/06249ccc-5c2a-401b-8e4b-6191104e0e2a-kube-api-access-mgvgr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n\" (UID: \"06249ccc-5c2a-401b-8e4b-6191104e0e2a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n"
Sep 11 11:56:24.989854 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:24.989822    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n"
Sep 11 11:56:25.002154 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:25.002134    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eef24a40-2739-487d-b111-66c0312a121c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq\" (UID: \"eef24a40-2739-487d-b111-66c0312a121c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq"
Sep 11 11:56:25.002237 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:25.002172    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eef24a40-2739-487d-b111-66c0312a121c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq\" (UID: \"eef24a40-2739-487d-b111-66c0312a121c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq"
Sep 11 11:56:25.002334 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:25.002313    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzh2h\" (UniqueName: \"kubernetes.io/projected/eef24a40-2739-487d-b111-66c0312a121c-kube-api-access-vzh2h\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq\" (UID: \"eef24a40-2739-487d-b111-66c0312a121c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq"
Sep 11 11:56:25.002526 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:25.002505    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eef24a40-2739-487d-b111-66c0312a121c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq\" (UID: \"eef24a40-2739-487d-b111-66c0312a121c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq"
Sep 11 11:56:25.002569 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:25.002513    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eef24a40-2739-487d-b111-66c0312a121c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq\" (UID: \"eef24a40-2739-487d-b111-66c0312a121c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq"
Sep 11 11:56:25.012322 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:25.012293    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzh2h\" (UniqueName: \"kubernetes.io/projected/eef24a40-2739-487d-b111-66c0312a121c-kube-api-access-vzh2h\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq\" (UID: \"eef24a40-2739-487d-b111-66c0312a121c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq"
Sep 11 11:56:25.074855 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:25.074830    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx"]
Sep 11 11:56:25.076543 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:56:25.076516    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca75ca39_1cd9_43d9_980f_e4da6dd51780.slice/crio-c8c2bce76586b2cf57533fd44f946e681664282f7f885a69ac1e851396a85189 WatchSource:0}: Error finding container c8c2bce76586b2cf57533fd44f946e681664282f7f885a69ac1e851396a85189: Status 404 returned error can't find the container with id c8c2bce76586b2cf57533fd44f946e681664282f7f885a69ac1e851396a85189
Sep 11 11:56:25.096704 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:25.096682    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq"
Sep 11 11:56:25.167146 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:25.167118    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh"]
Sep 11 11:56:25.170489 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:56:25.170458    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c483c7_01b3_4d1b_873a_f91e0ff20e6f.slice/crio-f4b90ff76967656e3b5f14f17f1f21912fc1c26f06dee99b896170de40375f77 WatchSource:0}: Error finding container f4b90ff76967656e3b5f14f17f1f21912fc1c26f06dee99b896170de40375f77: Status 404 returned error can't find the container with id f4b90ff76967656e3b5f14f17f1f21912fc1c26f06dee99b896170de40375f77
Sep 11 11:56:25.233740 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:25.233716    2569 generic.go:356] "Generic (PLEG): container finished" podID="ca75ca39-1cd9-43d9-980f-e4da6dd51780" containerID="033e6bb5b0bab5e8f0fad531c3b0f1704fff4b9724b8a5c0f344ae90e1d2a9b3" exitCode=0
Sep 11 11:56:25.233859 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:25.233799    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx" event={"ID":"ca75ca39-1cd9-43d9-980f-e4da6dd51780","Type":"ContainerDied","Data":"033e6bb5b0bab5e8f0fad531c3b0f1704fff4b9724b8a5c0f344ae90e1d2a9b3"}
Sep 11 11:56:25.233859 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:25.233850    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx" event={"ID":"ca75ca39-1cd9-43d9-980f-e4da6dd51780","Type":"ContainerStarted","Data":"c8c2bce76586b2cf57533fd44f946e681664282f7f885a69ac1e851396a85189"}
Sep 11 11:56:25.235237 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:25.235151    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh" event={"ID":"76c483c7-01b3-4d1b-873a-f91e0ff20e6f","Type":"ContainerStarted","Data":"f4b90ff76967656e3b5f14f17f1f21912fc1c26f06dee99b896170de40375f77"}
Sep 11 11:56:25.258762 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:25.258742    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n"]
Sep 11 11:56:25.260408 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:56:25.260379    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06249ccc_5c2a_401b_8e4b_6191104e0e2a.slice/crio-38f23548b93fbc0506bc4b4bfb3d7ea79b55a95da25f342f820f9304938dc754 WatchSource:0}: Error finding container 38f23548b93fbc0506bc4b4bfb3d7ea79b55a95da25f342f820f9304938dc754: Status 404 returned error can't find the container with id 38f23548b93fbc0506bc4b4bfb3d7ea79b55a95da25f342f820f9304938dc754
Sep 11 11:56:25.370699 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:25.370669    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq"]
Sep 11 11:56:25.417431 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:56:25.417407    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef24a40_2739_487d_b111_66c0312a121c.slice/crio-9e59bbce813f7efde43101bfb3b0c1cd1aacbbfaa00c430bb641e86289f976cf WatchSource:0}: Error finding container 9e59bbce813f7efde43101bfb3b0c1cd1aacbbfaa00c430bb641e86289f976cf: Status 404 returned error can't find the container with id 9e59bbce813f7efde43101bfb3b0c1cd1aacbbfaa00c430bb641e86289f976cf
Sep 11 11:56:25.532092 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:56:25.532058    2569 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef24a40_2739_487d_b111_66c0312a121c.slice/crio-conmon-b4eb5567aa2a1b375e050bed616ed22802534f53a682f5305759de50b031937f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef24a40_2739_487d_b111_66c0312a121c.slice/crio-b4eb5567aa2a1b375e050bed616ed22802534f53a682f5305759de50b031937f.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 11:56:26.243014 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:26.242938    2569 generic.go:356] "Generic (PLEG): container finished" podID="76c483c7-01b3-4d1b-873a-f91e0ff20e6f" containerID="cdca150ecfe136b7e03bceaaf59d37321a96955d3a8180e69d54ac685331beb5" exitCode=0
Sep 11 11:56:26.243181 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:26.243025    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh" event={"ID":"76c483c7-01b3-4d1b-873a-f91e0ff20e6f","Type":"ContainerDied","Data":"cdca150ecfe136b7e03bceaaf59d37321a96955d3a8180e69d54ac685331beb5"}
Sep 11 11:56:26.244736 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:26.244717    2569 generic.go:356] "Generic (PLEG): container finished" podID="eef24a40-2739-487d-b111-66c0312a121c" containerID="b4eb5567aa2a1b375e050bed616ed22802534f53a682f5305759de50b031937f" exitCode=0
Sep 11 11:56:26.244818 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:26.244794    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq" event={"ID":"eef24a40-2739-487d-b111-66c0312a121c","Type":"ContainerDied","Data":"b4eb5567aa2a1b375e050bed616ed22802534f53a682f5305759de50b031937f"}
Sep 11 11:56:26.244871 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:26.244855    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq" event={"ID":"eef24a40-2739-487d-b111-66c0312a121c","Type":"ContainerStarted","Data":"9e59bbce813f7efde43101bfb3b0c1cd1aacbbfaa00c430bb641e86289f976cf"}
Sep 11 11:56:26.246430 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:26.246373    2569 generic.go:356] "Generic (PLEG): container finished" podID="06249ccc-5c2a-401b-8e4b-6191104e0e2a" containerID="4aab6a9052f8ff03609340a50c92d5a1844a972b722a6b8108bd4e28be292993" exitCode=0
Sep 11 11:56:26.246502 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:26.246452    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n" event={"ID":"06249ccc-5c2a-401b-8e4b-6191104e0e2a","Type":"ContainerDied","Data":"4aab6a9052f8ff03609340a50c92d5a1844a972b722a6b8108bd4e28be292993"}
Sep 11 11:56:26.246502 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:26.246500    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n" event={"ID":"06249ccc-5c2a-401b-8e4b-6191104e0e2a","Type":"ContainerStarted","Data":"38f23548b93fbc0506bc4b4bfb3d7ea79b55a95da25f342f820f9304938dc754"}
Sep 11 11:56:27.253524 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:27.253499    2569 generic.go:356] "Generic (PLEG): container finished" podID="76c483c7-01b3-4d1b-873a-f91e0ff20e6f" containerID="76fe47d9fa71d543b46eec3c84c7e0d384ccbf3ab955f7f24a5da84cc7eb3b31" exitCode=0
Sep 11 11:56:27.253884 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:27.253577    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh" event={"ID":"76c483c7-01b3-4d1b-873a-f91e0ff20e6f","Type":"ContainerDied","Data":"76fe47d9fa71d543b46eec3c84c7e0d384ccbf3ab955f7f24a5da84cc7eb3b31"}
Sep 11 11:56:27.255603 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:27.255566    2569 generic.go:356] "Generic (PLEG): container finished" podID="ca75ca39-1cd9-43d9-980f-e4da6dd51780" containerID="cf2e200ef28297ac10dd60289c9c8ee3a3796c6ebfdf6b92ea066eafbc5ca165" exitCode=0
Sep 11 11:56:27.255692 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:27.255619    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx" event={"ID":"ca75ca39-1cd9-43d9-980f-e4da6dd51780","Type":"ContainerDied","Data":"cf2e200ef28297ac10dd60289c9c8ee3a3796c6ebfdf6b92ea066eafbc5ca165"}
Sep 11 11:56:28.268956 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:28.268925    2569 generic.go:356] "Generic (PLEG): container finished" podID="76c483c7-01b3-4d1b-873a-f91e0ff20e6f" containerID="11b0e5c9c9b95f51d678e42fe87d34f8be7a8d7d0879f3d96445f7bca03672a7" exitCode=0
Sep 11 11:56:28.269369 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:28.269016    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh" event={"ID":"76c483c7-01b3-4d1b-873a-f91e0ff20e6f","Type":"ContainerDied","Data":"11b0e5c9c9b95f51d678e42fe87d34f8be7a8d7d0879f3d96445f7bca03672a7"}
Sep 11 11:56:28.270995 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:28.270970    2569 generic.go:356] "Generic (PLEG): container finished" podID="ca75ca39-1cd9-43d9-980f-e4da6dd51780" containerID="4b46f1eb9a5ffb5841619871fe9e141cc0341478f099a6fa8a7ff81a2ed5779d" exitCode=0
Sep 11 11:56:28.271104 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:28.271050    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx" event={"ID":"ca75ca39-1cd9-43d9-980f-e4da6dd51780","Type":"ContainerDied","Data":"4b46f1eb9a5ffb5841619871fe9e141cc0341478f099a6fa8a7ff81a2ed5779d"}
Sep 11 11:56:29.277345 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.277318    2569 generic.go:356] "Generic (PLEG): container finished" podID="eef24a40-2739-487d-b111-66c0312a121c" containerID="b9082d34f82e339579c8c6f8d5ab9a83dd3a0fbae5477bcccbecfaaca7b82e3a" exitCode=0
Sep 11 11:56:29.277857 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.277396    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq" event={"ID":"eef24a40-2739-487d-b111-66c0312a121c","Type":"ContainerDied","Data":"b9082d34f82e339579c8c6f8d5ab9a83dd3a0fbae5477bcccbecfaaca7b82e3a"}
Sep 11 11:56:29.279346 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.279323    2569 generic.go:356] "Generic (PLEG): container finished" podID="06249ccc-5c2a-401b-8e4b-6191104e0e2a" containerID="0c3e8ca4ef81272753736cf37792d0da272b3ee4c8f36b3be7c5816609c00500" exitCode=0
Sep 11 11:56:29.279463 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.279399    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n" event={"ID":"06249ccc-5c2a-401b-8e4b-6191104e0e2a","Type":"ContainerDied","Data":"0c3e8ca4ef81272753736cf37792d0da272b3ee4c8f36b3be7c5816609c00500"}
Sep 11 11:56:29.409816 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.409792    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx"
Sep 11 11:56:29.523732 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.523712    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh"
Sep 11 11:56:29.539635 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.539562    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca75ca39-1cd9-43d9-980f-e4da6dd51780-bundle\") pod \"ca75ca39-1cd9-43d9-980f-e4da6dd51780\" (UID: \"ca75ca39-1cd9-43d9-980f-e4da6dd51780\") "
Sep 11 11:56:29.539635 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.539610    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-244bd\" (UniqueName: \"kubernetes.io/projected/ca75ca39-1cd9-43d9-980f-e4da6dd51780-kube-api-access-244bd\") pod \"ca75ca39-1cd9-43d9-980f-e4da6dd51780\" (UID: \"ca75ca39-1cd9-43d9-980f-e4da6dd51780\") "
Sep 11 11:56:29.539806 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.539678    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca75ca39-1cd9-43d9-980f-e4da6dd51780-util\") pod \"ca75ca39-1cd9-43d9-980f-e4da6dd51780\" (UID: \"ca75ca39-1cd9-43d9-980f-e4da6dd51780\") "
Sep 11 11:56:29.540080 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.540056    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca75ca39-1cd9-43d9-980f-e4da6dd51780-bundle" (OuterVolumeSpecName: "bundle") pod "ca75ca39-1cd9-43d9-980f-e4da6dd51780" (UID: "ca75ca39-1cd9-43d9-980f-e4da6dd51780"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:56:29.541571 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.541551    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca75ca39-1cd9-43d9-980f-e4da6dd51780-kube-api-access-244bd" (OuterVolumeSpecName: "kube-api-access-244bd") pod "ca75ca39-1cd9-43d9-980f-e4da6dd51780" (UID: "ca75ca39-1cd9-43d9-980f-e4da6dd51780"). InnerVolumeSpecName "kube-api-access-244bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:56:29.545334 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.545314    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca75ca39-1cd9-43d9-980f-e4da6dd51780-util" (OuterVolumeSpecName: "util") pod "ca75ca39-1cd9-43d9-980f-e4da6dd51780" (UID: "ca75ca39-1cd9-43d9-980f-e4da6dd51780"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:56:29.595319 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.595282    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f5bd46c46-8q5kr"]
Sep 11 11:56:29.595766 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.595714    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca75ca39-1cd9-43d9-980f-e4da6dd51780" containerName="util"
Sep 11 11:56:29.595766 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.595734    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca75ca39-1cd9-43d9-980f-e4da6dd51780" containerName="util"
Sep 11 11:56:29.595766 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.595747    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c483c7-01b3-4d1b-873a-f91e0ff20e6f" containerName="extract"
Sep 11 11:56:29.595766 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.595755    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c483c7-01b3-4d1b-873a-f91e0ff20e6f" containerName="extract"
Sep 11 11:56:29.596049 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.595771    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c483c7-01b3-4d1b-873a-f91e0ff20e6f" containerName="pull"
Sep 11 11:56:29.596049 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.595780    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c483c7-01b3-4d1b-873a-f91e0ff20e6f" containerName="pull"
Sep 11 11:56:29.596049 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.595793    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca75ca39-1cd9-43d9-980f-e4da6dd51780" containerName="extract"
Sep 11 11:56:29.596049 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.595800    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca75ca39-1cd9-43d9-980f-e4da6dd51780" containerName="extract"
Sep 11 11:56:29.596049 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.595817    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca75ca39-1cd9-43d9-980f-e4da6dd51780" containerName="pull"
Sep 11 11:56:29.596049 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.595825    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca75ca39-1cd9-43d9-980f-e4da6dd51780" containerName="pull"
Sep 11 11:56:29.596049 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.595849    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c483c7-01b3-4d1b-873a-f91e0ff20e6f" containerName="util"
Sep 11 11:56:29.596049 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.595856    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c483c7-01b3-4d1b-873a-f91e0ff20e6f" containerName="util"
Sep 11 11:56:29.596049 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.595965    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="76c483c7-01b3-4d1b-873a-f91e0ff20e6f" containerName="extract"
Sep 11 11:56:29.596049 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.595978    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca75ca39-1cd9-43d9-980f-e4da6dd51780" containerName="extract"
Sep 11 11:56:29.599055 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.599035    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.615770 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.615751    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f5bd46c46-8q5kr"]
Sep 11 11:56:29.640684 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.640657    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-bundle\") pod \"76c483c7-01b3-4d1b-873a-f91e0ff20e6f\" (UID: \"76c483c7-01b3-4d1b-873a-f91e0ff20e6f\") "
Sep 11 11:56:29.640850 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.640833    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9qpb\" (UniqueName: \"kubernetes.io/projected/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-kube-api-access-t9qpb\") pod \"76c483c7-01b3-4d1b-873a-f91e0ff20e6f\" (UID: \"76c483c7-01b3-4d1b-873a-f91e0ff20e6f\") "
Sep 11 11:56:29.640906 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.640889    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-util\") pod \"76c483c7-01b3-4d1b-873a-f91e0ff20e6f\" (UID: \"76c483c7-01b3-4d1b-873a-f91e0ff20e6f\") "
Sep 11 11:56:29.641023 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.641005    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7841f7e9-a8e7-4343-b973-50e696a373c1-console-oauth-config\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.641084 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.641037    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7841f7e9-a8e7-4343-b973-50e696a373c1-service-ca\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.641084 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.641055    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7841f7e9-a8e7-4343-b973-50e696a373c1-console-serving-cert\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.641084 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.641080    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7841f7e9-a8e7-4343-b973-50e696a373c1-oauth-serving-cert\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.641255 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.641147    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcrnk\" (UniqueName: \"kubernetes.io/projected/7841f7e9-a8e7-4343-b973-50e696a373c1-kube-api-access-mcrnk\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.641255 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.641132    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-bundle" (OuterVolumeSpecName: "bundle") pod "76c483c7-01b3-4d1b-873a-f91e0ff20e6f" (UID: "76c483c7-01b3-4d1b-873a-f91e0ff20e6f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:56:29.641255 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.641210    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7841f7e9-a8e7-4343-b973-50e696a373c1-trusted-ca-bundle\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.641394 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.641276    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7841f7e9-a8e7-4343-b973-50e696a373c1-console-config\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.641394 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.641315    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-bundle\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:56:29.641394 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.641326    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca75ca39-1cd9-43d9-980f-e4da6dd51780-util\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:56:29.641394 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.641335    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca75ca39-1cd9-43d9-980f-e4da6dd51780-bundle\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:56:29.641394 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.641344    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-244bd\" (UniqueName: \"kubernetes.io/projected/ca75ca39-1cd9-43d9-980f-e4da6dd51780-kube-api-access-244bd\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:56:29.642903 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.642876    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-kube-api-access-t9qpb" (OuterVolumeSpecName: "kube-api-access-t9qpb") pod "76c483c7-01b3-4d1b-873a-f91e0ff20e6f" (UID: "76c483c7-01b3-4d1b-873a-f91e0ff20e6f"). InnerVolumeSpecName "kube-api-access-t9qpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:56:29.646900 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.646880    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-util" (OuterVolumeSpecName: "util") pod "76c483c7-01b3-4d1b-873a-f91e0ff20e6f" (UID: "76c483c7-01b3-4d1b-873a-f91e0ff20e6f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:56:29.742220 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.742196    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7841f7e9-a8e7-4343-b973-50e696a373c1-trusted-ca-bundle\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.742306 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.742237    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7841f7e9-a8e7-4343-b973-50e696a373c1-console-config\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.742306 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.742265    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7841f7e9-a8e7-4343-b973-50e696a373c1-console-oauth-config\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.742306 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.742285    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7841f7e9-a8e7-4343-b973-50e696a373c1-service-ca\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.742306 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.742300    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7841f7e9-a8e7-4343-b973-50e696a373c1-console-serving-cert\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.742515 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.742323    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7841f7e9-a8e7-4343-b973-50e696a373c1-oauth-serving-cert\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.742515 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.742359    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcrnk\" (UniqueName: \"kubernetes.io/projected/7841f7e9-a8e7-4343-b973-50e696a373c1-kube-api-access-mcrnk\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.742515 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.742418    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9qpb\" (UniqueName: \"kubernetes.io/projected/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-kube-api-access-t9qpb\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:56:29.742515 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.742435    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76c483c7-01b3-4d1b-873a-f91e0ff20e6f-util\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:56:29.743011 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.742986    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7841f7e9-a8e7-4343-b973-50e696a373c1-console-config\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.743136 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.743119    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7841f7e9-a8e7-4343-b973-50e696a373c1-service-ca\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.743206 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.743160    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7841f7e9-a8e7-4343-b973-50e696a373c1-trusted-ca-bundle\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.743206 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.743169    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7841f7e9-a8e7-4343-b973-50e696a373c1-oauth-serving-cert\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.744672 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.744653    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7841f7e9-a8e7-4343-b973-50e696a373c1-console-serving-cert\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.744753 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.744701    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7841f7e9-a8e7-4343-b973-50e696a373c1-console-oauth-config\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.751691 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.751670    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcrnk\" (UniqueName: \"kubernetes.io/projected/7841f7e9-a8e7-4343-b973-50e696a373c1-kube-api-access-mcrnk\") pod \"console-7f5bd46c46-8q5kr\" (UID: \"7841f7e9-a8e7-4343-b973-50e696a373c1\") " pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:29.908733 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:29.908656    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:30.182381 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:30.182356    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f5bd46c46-8q5kr"]
Sep 11 11:56:30.183759 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:56:30.183725    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7841f7e9_a8e7_4343_b973_50e696a373c1.slice/crio-41fbb5fd7357211e6b8520fa5a53131fdd0e3990e533e4d5d6ecfc7a9827c0c5 WatchSource:0}: Error finding container 41fbb5fd7357211e6b8520fa5a53131fdd0e3990e533e4d5d6ecfc7a9827c0c5: Status 404 returned error can't find the container with id 41fbb5fd7357211e6b8520fa5a53131fdd0e3990e533e4d5d6ecfc7a9827c0c5
Sep 11 11:56:30.286414 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:30.286387    2569 generic.go:356] "Generic (PLEG): container finished" podID="eef24a40-2739-487d-b111-66c0312a121c" containerID="965b46f03ca729bc45ff0f72d8179ce16bfa0ed4628fb9ce368b6c8986bee73f" exitCode=0
Sep 11 11:56:30.286817 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:30.286472    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq" event={"ID":"eef24a40-2739-487d-b111-66c0312a121c","Type":"ContainerDied","Data":"965b46f03ca729bc45ff0f72d8179ce16bfa0ed4628fb9ce368b6c8986bee73f"}
Sep 11 11:56:30.289221 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:30.289198    2569 generic.go:356] "Generic (PLEG): container finished" podID="06249ccc-5c2a-401b-8e4b-6191104e0e2a" containerID="1d75fce6c4c41bc1c49edd96e1190a2335a857242e8bad6f4999121636ba5761" exitCode=0
Sep 11 11:56:30.289334 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:30.289283    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n" event={"ID":"06249ccc-5c2a-401b-8e4b-6191104e0e2a","Type":"ContainerDied","Data":"1d75fce6c4c41bc1c49edd96e1190a2335a857242e8bad6f4999121636ba5761"}
Sep 11 11:56:30.291335 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:30.291315    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx"
Sep 11 11:56:30.291441 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:30.291335    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6xxx" event={"ID":"ca75ca39-1cd9-43d9-980f-e4da6dd51780","Type":"ContainerDied","Data":"c8c2bce76586b2cf57533fd44f946e681664282f7f885a69ac1e851396a85189"}
Sep 11 11:56:30.291441 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:30.291375    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8c2bce76586b2cf57533fd44f946e681664282f7f885a69ac1e851396a85189"
Sep 11 11:56:30.293135 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:30.293110    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5bd46c46-8q5kr" event={"ID":"7841f7e9-a8e7-4343-b973-50e696a373c1","Type":"ContainerStarted","Data":"4ed8e092988067fd638c2ecaede1f82b6e581ef21b94278ee631823190ecc896"}
Sep 11 11:56:30.293239 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:30.293178    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5bd46c46-8q5kr" event={"ID":"7841f7e9-a8e7-4343-b973-50e696a373c1","Type":"ContainerStarted","Data":"41fbb5fd7357211e6b8520fa5a53131fdd0e3990e533e4d5d6ecfc7a9827c0c5"}
Sep 11 11:56:30.295391 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:30.295371    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh" event={"ID":"76c483c7-01b3-4d1b-873a-f91e0ff20e6f","Type":"ContainerDied","Data":"f4b90ff76967656e3b5f14f17f1f21912fc1c26f06dee99b896170de40375f77"}
Sep 11 11:56:30.295459 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:30.295403    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c96gwh"
Sep 11 11:56:30.295459 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:30.295408    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4b90ff76967656e3b5f14f17f1f21912fc1c26f06dee99b896170de40375f77"
Sep 11 11:56:30.352411 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:30.352369    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f5bd46c46-8q5kr" podStartSLOduration=1.352356185 podStartE2EDuration="1.352356185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:56:30.350321336 +0000 UTC m=+512.772189744" watchObservedRunningTime="2026-09-11 11:56:30.352356185 +0000 UTC m=+512.774224565"
Sep 11 11:56:31.449485 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.449460    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n"
Sep 11 11:56:31.452659 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.452642    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq"
Sep 11 11:56:31.453972 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.453954    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06249ccc-5c2a-401b-8e4b-6191104e0e2a-bundle\") pod \"06249ccc-5c2a-401b-8e4b-6191104e0e2a\" (UID: \"06249ccc-5c2a-401b-8e4b-6191104e0e2a\") "
Sep 11 11:56:31.454071 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.453988    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgvgr\" (UniqueName: \"kubernetes.io/projected/06249ccc-5c2a-401b-8e4b-6191104e0e2a-kube-api-access-mgvgr\") pod \"06249ccc-5c2a-401b-8e4b-6191104e0e2a\" (UID: \"06249ccc-5c2a-401b-8e4b-6191104e0e2a\") "
Sep 11 11:56:31.454071 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.454016    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06249ccc-5c2a-401b-8e4b-6191104e0e2a-util\") pod \"06249ccc-5c2a-401b-8e4b-6191104e0e2a\" (UID: \"06249ccc-5c2a-401b-8e4b-6191104e0e2a\") "
Sep 11 11:56:31.454071 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.454035    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eef24a40-2739-487d-b111-66c0312a121c-bundle\") pod \"eef24a40-2739-487d-b111-66c0312a121c\" (UID: \"eef24a40-2739-487d-b111-66c0312a121c\") "
Sep 11 11:56:31.454232 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.454090    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzh2h\" (UniqueName: \"kubernetes.io/projected/eef24a40-2739-487d-b111-66c0312a121c-kube-api-access-vzh2h\") pod \"eef24a40-2739-487d-b111-66c0312a121c\" (UID: \"eef24a40-2739-487d-b111-66c0312a121c\") "
Sep 11 11:56:31.454232 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.454116    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eef24a40-2739-487d-b111-66c0312a121c-util\") pod \"eef24a40-2739-487d-b111-66c0312a121c\" (UID: \"eef24a40-2739-487d-b111-66c0312a121c\") "
Sep 11 11:56:31.454508 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.454479    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06249ccc-5c2a-401b-8e4b-6191104e0e2a-bundle" (OuterVolumeSpecName: "bundle") pod "06249ccc-5c2a-401b-8e4b-6191104e0e2a" (UID: "06249ccc-5c2a-401b-8e4b-6191104e0e2a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:56:31.454884 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.454858    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef24a40-2739-487d-b111-66c0312a121c-bundle" (OuterVolumeSpecName: "bundle") pod "eef24a40-2739-487d-b111-66c0312a121c" (UID: "eef24a40-2739-487d-b111-66c0312a121c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:56:31.456146 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.456120    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06249ccc-5c2a-401b-8e4b-6191104e0e2a-kube-api-access-mgvgr" (OuterVolumeSpecName: "kube-api-access-mgvgr") pod "06249ccc-5c2a-401b-8e4b-6191104e0e2a" (UID: "06249ccc-5c2a-401b-8e4b-6191104e0e2a"). InnerVolumeSpecName "kube-api-access-mgvgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:56:31.456223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.456164    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef24a40-2739-487d-b111-66c0312a121c-kube-api-access-vzh2h" (OuterVolumeSpecName: "kube-api-access-vzh2h") pod "eef24a40-2739-487d-b111-66c0312a121c" (UID: "eef24a40-2739-487d-b111-66c0312a121c"). InnerVolumeSpecName "kube-api-access-vzh2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:56:31.459348 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.459317    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06249ccc-5c2a-401b-8e4b-6191104e0e2a-util" (OuterVolumeSpecName: "util") pod "06249ccc-5c2a-401b-8e4b-6191104e0e2a" (UID: "06249ccc-5c2a-401b-8e4b-6191104e0e2a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:56:31.461099 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.461076    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef24a40-2739-487d-b111-66c0312a121c-util" (OuterVolumeSpecName: "util") pod "eef24a40-2739-487d-b111-66c0312a121c" (UID: "eef24a40-2739-487d-b111-66c0312a121c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 11:56:31.554798 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.554736    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzh2h\" (UniqueName: \"kubernetes.io/projected/eef24a40-2739-487d-b111-66c0312a121c-kube-api-access-vzh2h\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:56:31.554798 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.554760    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eef24a40-2739-487d-b111-66c0312a121c-util\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:56:31.554798 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.554769    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06249ccc-5c2a-401b-8e4b-6191104e0e2a-bundle\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:56:31.554798 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.554777    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgvgr\" (UniqueName: \"kubernetes.io/projected/06249ccc-5c2a-401b-8e4b-6191104e0e2a-kube-api-access-mgvgr\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:56:31.554798 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.554785    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06249ccc-5c2a-401b-8e4b-6191104e0e2a-util\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:56:31.554798 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:31.554798    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eef24a40-2739-487d-b111-66c0312a121c-bundle\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:56:32.308683 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:32.308647    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq" event={"ID":"eef24a40-2739-487d-b111-66c0312a121c","Type":"ContainerDied","Data":"9e59bbce813f7efde43101bfb3b0c1cd1aacbbfaa00c430bb641e86289f976cf"}
Sep 11 11:56:32.308836 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:32.308705    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e59bbce813f7efde43101bfb3b0c1cd1aacbbfaa00c430bb641e86289f976cf"
Sep 11 11:56:32.308836 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:32.308652    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9pxcq"
Sep 11 11:56:32.310954 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:32.310926    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n" event={"ID":"06249ccc-5c2a-401b-8e4b-6191104e0e2a","Type":"ContainerDied","Data":"38f23548b93fbc0506bc4b4bfb3d7ea79b55a95da25f342f820f9304938dc754"}
Sep 11 11:56:32.311082 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:32.310960    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qws5n"
Sep 11 11:56:32.311082 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:32.310967    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38f23548b93fbc0506bc4b4bfb3d7ea79b55a95da25f342f820f9304938dc754"
Sep 11 11:56:39.909230 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:39.909198    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:39.909643 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:39.909255    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:39.914141 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:39.914113    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:40.356159 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:40.356121    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f5bd46c46-8q5kr"
Sep 11 11:56:40.447040 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:56:40.447012    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-98b75cfb6-jjbrg"]
Sep 11 11:57:05.466274 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.466158    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-98b75cfb6-jjbrg" podUID="4197d176-a6c4-490f-8353-0f37403bf815" containerName="console" containerID="cri-o://96d6ab41bdea57ca92362b23f8c378020f3bb608cdb01b1eb836e49820c13a52" gracePeriod=15
Sep 11 11:57:05.711982 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.711955    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:57:05.831379 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.831349    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-trusted-ca-bundle\") pod \"4197d176-a6c4-490f-8353-0f37403bf815\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") "
Sep 11 11:57:05.831503 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.831419    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-console-config\") pod \"4197d176-a6c4-490f-8353-0f37403bf815\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") "
Sep 11 11:57:05.831503 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.831441    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4197d176-a6c4-490f-8353-0f37403bf815-console-oauth-config\") pod \"4197d176-a6c4-490f-8353-0f37403bf815\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") "
Sep 11 11:57:05.831503 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.831465    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4197d176-a6c4-490f-8353-0f37403bf815-console-serving-cert\") pod \"4197d176-a6c4-490f-8353-0f37403bf815\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") "
Sep 11 11:57:05.831503 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.831487    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-service-ca\") pod \"4197d176-a6c4-490f-8353-0f37403bf815\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") "
Sep 11 11:57:05.831706 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.831562    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-oauth-serving-cert\") pod \"4197d176-a6c4-490f-8353-0f37403bf815\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") "
Sep 11 11:57:05.831706 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.831625    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkn48\" (UniqueName: \"kubernetes.io/projected/4197d176-a6c4-490f-8353-0f37403bf815-kube-api-access-lkn48\") pod \"4197d176-a6c4-490f-8353-0f37403bf815\" (UID: \"4197d176-a6c4-490f-8353-0f37403bf815\") "
Sep 11 11:57:05.831931 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.831774    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4197d176-a6c4-490f-8353-0f37403bf815" (UID: "4197d176-a6c4-490f-8353-0f37403bf815"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 11:57:05.832038 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.831954    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-console-config" (OuterVolumeSpecName: "console-config") pod "4197d176-a6c4-490f-8353-0f37403bf815" (UID: "4197d176-a6c4-490f-8353-0f37403bf815"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 11:57:05.832038 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.831962    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-service-ca" (OuterVolumeSpecName: "service-ca") pod "4197d176-a6c4-490f-8353-0f37403bf815" (UID: "4197d176-a6c4-490f-8353-0f37403bf815"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 11:57:05.832038 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.831993    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4197d176-a6c4-490f-8353-0f37403bf815" (UID: "4197d176-a6c4-490f-8353-0f37403bf815"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 11:57:05.833767 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.833740    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4197d176-a6c4-490f-8353-0f37403bf815-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4197d176-a6c4-490f-8353-0f37403bf815" (UID: "4197d176-a6c4-490f-8353-0f37403bf815"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 11:57:05.833767 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.833764    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4197d176-a6c4-490f-8353-0f37403bf815-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4197d176-a6c4-490f-8353-0f37403bf815" (UID: "4197d176-a6c4-490f-8353-0f37403bf815"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 11:57:05.833940 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.833783    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4197d176-a6c4-490f-8353-0f37403bf815-kube-api-access-lkn48" (OuterVolumeSpecName: "kube-api-access-lkn48") pod "4197d176-a6c4-490f-8353-0f37403bf815" (UID: "4197d176-a6c4-490f-8353-0f37403bf815"). InnerVolumeSpecName "kube-api-access-lkn48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:57:05.933067 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.933042    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-oauth-serving-cert\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:57:05.933067 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.933063    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkn48\" (UniqueName: \"kubernetes.io/projected/4197d176-a6c4-490f-8353-0f37403bf815-kube-api-access-lkn48\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:57:05.933204 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.933072    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-trusted-ca-bundle\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:57:05.933204 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.933082    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-console-config\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:57:05.933204 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.933090    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4197d176-a6c4-490f-8353-0f37403bf815-console-oauth-config\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:57:05.933204 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.933099    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4197d176-a6c4-490f-8353-0f37403bf815-console-serving-cert\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:57:05.933204 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:05.933107    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4197d176-a6c4-490f-8353-0f37403bf815-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:57:06.490906 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:06.490871    2569 generic.go:356] "Generic (PLEG): container finished" podID="4197d176-a6c4-490f-8353-0f37403bf815" containerID="96d6ab41bdea57ca92362b23f8c378020f3bb608cdb01b1eb836e49820c13a52" exitCode=0
Sep 11 11:57:06.491300 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:06.490915    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-98b75cfb6-jjbrg"
Sep 11 11:57:06.491300 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:06.490960    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-98b75cfb6-jjbrg" event={"ID":"4197d176-a6c4-490f-8353-0f37403bf815","Type":"ContainerDied","Data":"96d6ab41bdea57ca92362b23f8c378020f3bb608cdb01b1eb836e49820c13a52"}
Sep 11 11:57:06.491300 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:06.491012    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-98b75cfb6-jjbrg" event={"ID":"4197d176-a6c4-490f-8353-0f37403bf815","Type":"ContainerDied","Data":"91f9e6b98db8f13bc7f917be88c49f981e1c59d89ce4bcc2553806d2e421885d"}
Sep 11 11:57:06.491300 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:06.491035    2569 scope.go:117] "RemoveContainer" containerID="96d6ab41bdea57ca92362b23f8c378020f3bb608cdb01b1eb836e49820c13a52"
Sep 11 11:57:06.499922 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:06.499906    2569 scope.go:117] "RemoveContainer" containerID="96d6ab41bdea57ca92362b23f8c378020f3bb608cdb01b1eb836e49820c13a52"
Sep 11 11:57:06.500202 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:57:06.500184    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d6ab41bdea57ca92362b23f8c378020f3bb608cdb01b1eb836e49820c13a52\": container with ID starting with 96d6ab41bdea57ca92362b23f8c378020f3bb608cdb01b1eb836e49820c13a52 not found: ID does not exist" containerID="96d6ab41bdea57ca92362b23f8c378020f3bb608cdb01b1eb836e49820c13a52"
Sep 11 11:57:06.500249 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:06.500211    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d6ab41bdea57ca92362b23f8c378020f3bb608cdb01b1eb836e49820c13a52"} err="failed to get container status \"96d6ab41bdea57ca92362b23f8c378020f3bb608cdb01b1eb836e49820c13a52\": rpc error: code = NotFound desc = could not find container \"96d6ab41bdea57ca92362b23f8c378020f3bb608cdb01b1eb836e49820c13a52\": container with ID starting with 96d6ab41bdea57ca92362b23f8c378020f3bb608cdb01b1eb836e49820c13a52 not found: ID does not exist"
Sep 11 11:57:06.513239 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:06.513214    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-98b75cfb6-jjbrg"]
Sep 11 11:57:06.519663 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:06.519643    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-98b75cfb6-jjbrg"]
Sep 11 11:57:08.070709 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:08.070677    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4197d176-a6c4-490f-8353-0f37403bf815" path="/var/lib/kubelet/pods/4197d176-a6c4-490f-8353-0f37403bf815/volumes"
Sep 11 11:57:34.960749 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.960713    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6shqq"]
Sep 11 11:57:34.961111 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.961049    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef24a40-2739-487d-b111-66c0312a121c" containerName="pull"
Sep 11 11:57:34.961111 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.961059    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef24a40-2739-487d-b111-66c0312a121c" containerName="pull"
Sep 11 11:57:34.961111 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.961067    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef24a40-2739-487d-b111-66c0312a121c" containerName="extract"
Sep 11 11:57:34.961111 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.961073    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef24a40-2739-487d-b111-66c0312a121c" containerName="extract"
Sep 11 11:57:34.961111 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.961081    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06249ccc-5c2a-401b-8e4b-6191104e0e2a" containerName="pull"
Sep 11 11:57:34.961111 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.961087    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="06249ccc-5c2a-401b-8e4b-6191104e0e2a" containerName="pull"
Sep 11 11:57:34.961111 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.961100    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06249ccc-5c2a-401b-8e4b-6191104e0e2a" containerName="extract"
Sep 11 11:57:34.961111 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.961105    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="06249ccc-5c2a-401b-8e4b-6191104e0e2a" containerName="extract"
Sep 11 11:57:34.961111 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.961114    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06249ccc-5c2a-401b-8e4b-6191104e0e2a" containerName="util"
Sep 11 11:57:34.961379 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.961120    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="06249ccc-5c2a-401b-8e4b-6191104e0e2a" containerName="util"
Sep 11 11:57:34.961379 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.961129    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef24a40-2739-487d-b111-66c0312a121c" containerName="util"
Sep 11 11:57:34.961379 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.961135    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef24a40-2739-487d-b111-66c0312a121c" containerName="util"
Sep 11 11:57:34.961379 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.961142    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4197d176-a6c4-490f-8353-0f37403bf815" containerName="console"
Sep 11 11:57:34.961379 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.961147    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4197d176-a6c4-490f-8353-0f37403bf815" containerName="console"
Sep 11 11:57:34.961379 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.961196    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef24a40-2739-487d-b111-66c0312a121c" containerName="extract"
Sep 11 11:57:34.961379 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.961206    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4197d176-a6c4-490f-8353-0f37403bf815" containerName="console"
Sep 11 11:57:34.961379 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.961212    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="06249ccc-5c2a-401b-8e4b-6191104e0e2a" containerName="extract"
Sep 11 11:57:34.964247 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.964231    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-6shqq"
Sep 11 11:57:34.967377 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.967356    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 11:57:34.967503 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.967356    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 11:57:34.968069 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.968054    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 11:57:34.968069 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.968065    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h7kcd\""
Sep 11 11:57:34.979201 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:34.979178    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6shqq"]
Sep 11 11:57:35.065568 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:35.065532    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6shqq"]
Sep 11 11:57:35.146486 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:35.146456    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v24w\" (UniqueName: \"kubernetes.io/projected/c348f02e-f332-46a8-8526-1aff01a10a6a-kube-api-access-6v24w\") pod \"limitador-limitador-8588749977-6shqq\" (UID: \"c348f02e-f332-46a8-8526-1aff01a10a6a\") " pod="kuadrant-system/limitador-limitador-8588749977-6shqq"
Sep 11 11:57:35.146657 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:35.146534    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c348f02e-f332-46a8-8526-1aff01a10a6a-config-file\") pod \"limitador-limitador-8588749977-6shqq\" (UID: \"c348f02e-f332-46a8-8526-1aff01a10a6a\") " pod="kuadrant-system/limitador-limitador-8588749977-6shqq"
Sep 11 11:57:35.247112 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:35.247079    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v24w\" (UniqueName: \"kubernetes.io/projected/c348f02e-f332-46a8-8526-1aff01a10a6a-kube-api-access-6v24w\") pod \"limitador-limitador-8588749977-6shqq\" (UID: \"c348f02e-f332-46a8-8526-1aff01a10a6a\") " pod="kuadrant-system/limitador-limitador-8588749977-6shqq"
Sep 11 11:57:35.247262 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:35.247126    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c348f02e-f332-46a8-8526-1aff01a10a6a-config-file\") pod \"limitador-limitador-8588749977-6shqq\" (UID: \"c348f02e-f332-46a8-8526-1aff01a10a6a\") " pod="kuadrant-system/limitador-limitador-8588749977-6shqq"
Sep 11 11:57:35.247717 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:35.247699    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c348f02e-f332-46a8-8526-1aff01a10a6a-config-file\") pod \"limitador-limitador-8588749977-6shqq\" (UID: \"c348f02e-f332-46a8-8526-1aff01a10a6a\") " pod="kuadrant-system/limitador-limitador-8588749977-6shqq"
Sep 11 11:57:35.257515 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:35.257493    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v24w\" (UniqueName: \"kubernetes.io/projected/c348f02e-f332-46a8-8526-1aff01a10a6a-kube-api-access-6v24w\") pod \"limitador-limitador-8588749977-6shqq\" (UID: \"c348f02e-f332-46a8-8526-1aff01a10a6a\") " pod="kuadrant-system/limitador-limitador-8588749977-6shqq"
Sep 11 11:57:35.274482 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:35.274459    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-6shqq"
Sep 11 11:57:35.587344 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:57:35.587315    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc348f02e_f332_46a8_8526_1aff01a10a6a.slice/crio-ba6c2d7b6d8cb0ab989c20da15077365119bac62aab7e3986c369beae7e9ee83 WatchSource:0}: Error finding container ba6c2d7b6d8cb0ab989c20da15077365119bac62aab7e3986c369beae7e9ee83: Status 404 returned error can't find the container with id ba6c2d7b6d8cb0ab989c20da15077365119bac62aab7e3986c369beae7e9ee83
Sep 11 11:57:35.591446 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:35.591420    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6shqq"]
Sep 11 11:57:35.641304 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:35.641278    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-6shqq" event={"ID":"c348f02e-f332-46a8-8526-1aff01a10a6a","Type":"ContainerStarted","Data":"ba6c2d7b6d8cb0ab989c20da15077365119bac62aab7e3986c369beae7e9ee83"}
Sep 11 11:57:35.952186 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:35.952117    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6v5ln"]
Sep 11 11:57:35.956820 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:35.956800    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-6v5ln"
Sep 11 11:57:35.959764 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:35.959738    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pq2dp\""
Sep 11 11:57:35.969891 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:35.969865    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6v5ln"]
Sep 11 11:57:36.054354 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:36.054330    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8znk\" (UniqueName: \"kubernetes.io/projected/d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e-kube-api-access-f8znk\") pod \"authorino-7bcfd8f9cd-6v5ln\" (UID: \"d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e\") " pod="kuadrant-system/authorino-7bcfd8f9cd-6v5ln"
Sep 11 11:57:36.155397 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:36.155367    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8znk\" (UniqueName: \"kubernetes.io/projected/d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e-kube-api-access-f8znk\") pod \"authorino-7bcfd8f9cd-6v5ln\" (UID: \"d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e\") " pod="kuadrant-system/authorino-7bcfd8f9cd-6v5ln"
Sep 11 11:57:36.169117 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:36.169088    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8znk\" (UniqueName: \"kubernetes.io/projected/d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e-kube-api-access-f8znk\") pod \"authorino-7bcfd8f9cd-6v5ln\" (UID: \"d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e\") " pod="kuadrant-system/authorino-7bcfd8f9cd-6v5ln"
Sep 11 11:57:36.265892 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:36.265866    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-6v5ln"
Sep 11 11:57:36.541503 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:36.541420    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6v5ln"]
Sep 11 11:57:36.544795 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:57:36.544765    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd17d1fbe_4c9e_47e7_bdd3_14783e7e4f5e.slice/crio-a88eff33676eac18a458f978c6ab14e2d1acd0cc84cf6b777a2836894dfb27aa WatchSource:0}: Error finding container a88eff33676eac18a458f978c6ab14e2d1acd0cc84cf6b777a2836894dfb27aa: Status 404 returned error can't find the container with id a88eff33676eac18a458f978c6ab14e2d1acd0cc84cf6b777a2836894dfb27aa
Sep 11 11:57:36.649290 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:36.649252    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-6v5ln" event={"ID":"d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e","Type":"ContainerStarted","Data":"a88eff33676eac18a458f978c6ab14e2d1acd0cc84cf6b777a2836894dfb27aa"}
Sep 11 11:57:38.664001 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:38.663957    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-6shqq" event={"ID":"c348f02e-f332-46a8-8526-1aff01a10a6a","Type":"ContainerStarted","Data":"ad5aa32a1f175479909c4594b27c7f002ed9dea2aac2b52dc88a0cd209944d10"}
Sep 11 11:57:38.664435 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:38.664099    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-6shqq"
Sep 11 11:57:38.684277 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:38.684225    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-6shqq" podStartSLOduration=2.42511489 podStartE2EDuration="4.68420662s" totalImagesPullingTime="2.25909173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:57:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:57:38.683123364 +0000 UTC m=+581.104991761" watchObservedRunningTime="2026-09-11 11:57:38.68420662 +0000 UTC m=+581.106075010"
Sep 11 11:57:38.943079 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:38.943003    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6v5ln"]
Sep 11 11:57:39.670910 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:39.670866    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-6v5ln" event={"ID":"d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e","Type":"ContainerStarted","Data":"7f606f31367cf93f2c9baa7d7082cda3922893e46798bf074db924d2160fffb4"}
Sep 11 11:57:39.690022 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:39.689978    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-6v5ln" podStartSLOduration=2.443616992 podStartE2EDuration="4.689963988s" totalImagesPullingTime="2.246346996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:57:39.688312819 +0000 UTC m=+582.110181198" watchObservedRunningTime="2026-09-11 11:57:39.689963988 +0000 UTC m=+582.111832367"
Sep 11 11:57:40.676748 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:40.676695    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-6v5ln" podUID="d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e" containerName="authorino" containerID="cri-o://7f606f31367cf93f2c9baa7d7082cda3922893e46798bf074db924d2160fffb4" gracePeriod=30
Sep 11 11:57:40.924402 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:40.924378    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-6v5ln"
Sep 11 11:57:40.997707 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:40.997685    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8znk\" (UniqueName: \"kubernetes.io/projected/d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e-kube-api-access-f8znk\") pod \"d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e\" (UID: \"d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e\") "
Sep 11 11:57:40.999793 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:40.999763    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e-kube-api-access-f8znk" (OuterVolumeSpecName: "kube-api-access-f8znk") pod "d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e" (UID: "d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e"). InnerVolumeSpecName "kube-api-access-f8znk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:57:41.098895 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:41.098874    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8znk\" (UniqueName: \"kubernetes.io/projected/d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e-kube-api-access-f8znk\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:57:41.682965 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:41.682937    2569 generic.go:356] "Generic (PLEG): container finished" podID="d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e" containerID="7f606f31367cf93f2c9baa7d7082cda3922893e46798bf074db924d2160fffb4" exitCode=0
Sep 11 11:57:41.683451 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:41.682982    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-6v5ln"
Sep 11 11:57:41.683451 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:41.683026    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-6v5ln" event={"ID":"d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e","Type":"ContainerDied","Data":"7f606f31367cf93f2c9baa7d7082cda3922893e46798bf074db924d2160fffb4"}
Sep 11 11:57:41.683451 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:41.683081    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-6v5ln" event={"ID":"d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e","Type":"ContainerDied","Data":"a88eff33676eac18a458f978c6ab14e2d1acd0cc84cf6b777a2836894dfb27aa"}
Sep 11 11:57:41.683451 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:41.683108    2569 scope.go:117] "RemoveContainer" containerID="7f606f31367cf93f2c9baa7d7082cda3922893e46798bf074db924d2160fffb4"
Sep 11 11:57:41.692628 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:41.692610    2569 scope.go:117] "RemoveContainer" containerID="7f606f31367cf93f2c9baa7d7082cda3922893e46798bf074db924d2160fffb4"
Sep 11 11:57:41.692938 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:57:41.692917    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f606f31367cf93f2c9baa7d7082cda3922893e46798bf074db924d2160fffb4\": container with ID starting with 7f606f31367cf93f2c9baa7d7082cda3922893e46798bf074db924d2160fffb4 not found: ID does not exist" containerID="7f606f31367cf93f2c9baa7d7082cda3922893e46798bf074db924d2160fffb4"
Sep 11 11:57:41.692983 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:41.692948    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f606f31367cf93f2c9baa7d7082cda3922893e46798bf074db924d2160fffb4"} err="failed to get container status \"7f606f31367cf93f2c9baa7d7082cda3922893e46798bf074db924d2160fffb4\": rpc error: code = NotFound desc = could not find container \"7f606f31367cf93f2c9baa7d7082cda3922893e46798bf074db924d2160fffb4\": container with ID starting with 7f606f31367cf93f2c9baa7d7082cda3922893e46798bf074db924d2160fffb4 not found: ID does not exist"
Sep 11 11:57:41.705067 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:41.705042    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6v5ln"]
Sep 11 11:57:41.711225 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:41.711206    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-6v5ln"]
Sep 11 11:57:42.069806 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:42.069777    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e" path="/var/lib/kubelet/pods/d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e/volumes"
Sep 11 11:57:49.094174 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.094140    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6shqq"]
Sep 11 11:57:49.094572 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.094346    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-6shqq" podUID="c348f02e-f332-46a8-8526-1aff01a10a6a" containerName="limitador" containerID="cri-o://ad5aa32a1f175479909c4594b27c7f002ed9dea2aac2b52dc88a0cd209944d10" gracePeriod=30
Sep 11 11:57:49.095118 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.095018    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-6shqq"
Sep 11 11:57:49.630917 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.630891    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-6shqq"
Sep 11 11:57:49.665252 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.665196    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v24w\" (UniqueName: \"kubernetes.io/projected/c348f02e-f332-46a8-8526-1aff01a10a6a-kube-api-access-6v24w\") pod \"c348f02e-f332-46a8-8526-1aff01a10a6a\" (UID: \"c348f02e-f332-46a8-8526-1aff01a10a6a\") "
Sep 11 11:57:49.665345 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.665330    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c348f02e-f332-46a8-8526-1aff01a10a6a-config-file\") pod \"c348f02e-f332-46a8-8526-1aff01a10a6a\" (UID: \"c348f02e-f332-46a8-8526-1aff01a10a6a\") "
Sep 11 11:57:49.665704 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.665676    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c348f02e-f332-46a8-8526-1aff01a10a6a-config-file" (OuterVolumeSpecName: "config-file") pod "c348f02e-f332-46a8-8526-1aff01a10a6a" (UID: "c348f02e-f332-46a8-8526-1aff01a10a6a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 11:57:49.667303 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.667276    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c348f02e-f332-46a8-8526-1aff01a10a6a-kube-api-access-6v24w" (OuterVolumeSpecName: "kube-api-access-6v24w") pod "c348f02e-f332-46a8-8526-1aff01a10a6a" (UID: "c348f02e-f332-46a8-8526-1aff01a10a6a"). InnerVolumeSpecName "kube-api-access-6v24w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:57:49.726328 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.726298    2569 generic.go:356] "Generic (PLEG): container finished" podID="c348f02e-f332-46a8-8526-1aff01a10a6a" containerID="ad5aa32a1f175479909c4594b27c7f002ed9dea2aac2b52dc88a0cd209944d10" exitCode=0
Sep 11 11:57:49.726426 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.726353    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-6shqq"
Sep 11 11:57:49.726426 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.726377    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-6shqq" event={"ID":"c348f02e-f332-46a8-8526-1aff01a10a6a","Type":"ContainerDied","Data":"ad5aa32a1f175479909c4594b27c7f002ed9dea2aac2b52dc88a0cd209944d10"}
Sep 11 11:57:49.726498 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.726445    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-6shqq" event={"ID":"c348f02e-f332-46a8-8526-1aff01a10a6a","Type":"ContainerDied","Data":"ba6c2d7b6d8cb0ab989c20da15077365119bac62aab7e3986c369beae7e9ee83"}
Sep 11 11:57:49.726498 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.726468    2569 scope.go:117] "RemoveContainer" containerID="ad5aa32a1f175479909c4594b27c7f002ed9dea2aac2b52dc88a0cd209944d10"
Sep 11 11:57:49.734977 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.734961    2569 scope.go:117] "RemoveContainer" containerID="ad5aa32a1f175479909c4594b27c7f002ed9dea2aac2b52dc88a0cd209944d10"
Sep 11 11:57:49.735219 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:57:49.735202    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5aa32a1f175479909c4594b27c7f002ed9dea2aac2b52dc88a0cd209944d10\": container with ID starting with ad5aa32a1f175479909c4594b27c7f002ed9dea2aac2b52dc88a0cd209944d10 not found: ID does not exist" containerID="ad5aa32a1f175479909c4594b27c7f002ed9dea2aac2b52dc88a0cd209944d10"
Sep 11 11:57:49.735270 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.735228    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5aa32a1f175479909c4594b27c7f002ed9dea2aac2b52dc88a0cd209944d10"} err="failed to get container status \"ad5aa32a1f175479909c4594b27c7f002ed9dea2aac2b52dc88a0cd209944d10\": rpc error: code = NotFound desc = could not find container \"ad5aa32a1f175479909c4594b27c7f002ed9dea2aac2b52dc88a0cd209944d10\": container with ID starting with ad5aa32a1f175479909c4594b27c7f002ed9dea2aac2b52dc88a0cd209944d10 not found: ID does not exist"
Sep 11 11:57:49.749001 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.748978    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6shqq"]
Sep 11 11:57:49.753158 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.753138    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6shqq"]
Sep 11 11:57:49.766490 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.766472    2569 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c348f02e-f332-46a8-8526-1aff01a10a6a-config-file\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:57:49.766490 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:49.766490    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v24w\" (UniqueName: \"kubernetes.io/projected/c348f02e-f332-46a8-8526-1aff01a10a6a-kube-api-access-6v24w\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:57:50.069833 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:50.069806    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c348f02e-f332-46a8-8526-1aff01a10a6a" path="/var/lib/kubelet/pods/c348f02e-f332-46a8-8526-1aff01a10a6a/volumes"
Sep 11 11:57:51.456921 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.456877    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-t2bhm"]
Sep 11 11:57:51.457395 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.457212    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c348f02e-f332-46a8-8526-1aff01a10a6a" containerName="limitador"
Sep 11 11:57:51.457395 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.457224    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c348f02e-f332-46a8-8526-1aff01a10a6a" containerName="limitador"
Sep 11 11:57:51.457395 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.457241    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e" containerName="authorino"
Sep 11 11:57:51.457395 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.457247    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e" containerName="authorino"
Sep 11 11:57:51.457395 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.457324    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c348f02e-f332-46a8-8526-1aff01a10a6a" containerName="limitador"
Sep 11 11:57:51.457395 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.457335    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d17d1fbe-4c9e-47e7-bdd3-14783e7e4f5e" containerName="authorino"
Sep 11 11:57:51.461930 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.461908    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t2bhm"
Sep 11 11:57:51.464479 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.464459    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 11 11:57:51.464611 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.464548    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 11:57:51.465278 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.465261    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 11:57:51.465384 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.465275    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-f5sd4\""
Sep 11 11:57:51.469556 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.469527    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-t2bhm"]
Sep 11 11:57:51.479043 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.479003    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsf5w\" (UniqueName: \"kubernetes.io/projected/3c7a3895-ea08-4b99-a07e-21e7fb8667a6-kube-api-access-qsf5w\") pod \"postgres-69fb4594cf-t2bhm\" (UID: \"3c7a3895-ea08-4b99-a07e-21e7fb8667a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t2bhm"
Sep 11 11:57:51.479133 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.479069    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3c7a3895-ea08-4b99-a07e-21e7fb8667a6-data\") pod \"postgres-69fb4594cf-t2bhm\" (UID: \"3c7a3895-ea08-4b99-a07e-21e7fb8667a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t2bhm"
Sep 11 11:57:51.579562 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.579539    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3c7a3895-ea08-4b99-a07e-21e7fb8667a6-data\") pod \"postgres-69fb4594cf-t2bhm\" (UID: \"3c7a3895-ea08-4b99-a07e-21e7fb8667a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t2bhm"
Sep 11 11:57:51.579707 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.579691    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsf5w\" (UniqueName: \"kubernetes.io/projected/3c7a3895-ea08-4b99-a07e-21e7fb8667a6-kube-api-access-qsf5w\") pod \"postgres-69fb4594cf-t2bhm\" (UID: \"3c7a3895-ea08-4b99-a07e-21e7fb8667a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t2bhm"
Sep 11 11:57:51.579928 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.579903    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3c7a3895-ea08-4b99-a07e-21e7fb8667a6-data\") pod \"postgres-69fb4594cf-t2bhm\" (UID: \"3c7a3895-ea08-4b99-a07e-21e7fb8667a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t2bhm"
Sep 11 11:57:51.595071 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.595044    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsf5w\" (UniqueName: \"kubernetes.io/projected/3c7a3895-ea08-4b99-a07e-21e7fb8667a6-kube-api-access-qsf5w\") pod \"postgres-69fb4594cf-t2bhm\" (UID: \"3c7a3895-ea08-4b99-a07e-21e7fb8667a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t2bhm"
Sep 11 11:57:51.773992 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:51.773974    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t2bhm"
Sep 11 11:57:52.046185 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:52.046105    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-t2bhm"]
Sep 11 11:57:52.049194 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:57:52.049163    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7a3895_ea08_4b99_a07e_21e7fb8667a6.slice/crio-612cb9da790a3c68b143aefe2fe7e7a4eefd1837a999cb2e1e376488ef9dc51e WatchSource:0}: Error finding container 612cb9da790a3c68b143aefe2fe7e7a4eefd1837a999cb2e1e376488ef9dc51e: Status 404 returned error can't find the container with id 612cb9da790a3c68b143aefe2fe7e7a4eefd1837a999cb2e1e376488ef9dc51e
Sep 11 11:57:52.745887 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:52.745844    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t2bhm" event={"ID":"3c7a3895-ea08-4b99-a07e-21e7fb8667a6","Type":"ContainerStarted","Data":"612cb9da790a3c68b143aefe2fe7e7a4eefd1837a999cb2e1e376488ef9dc51e"}
Sep 11 11:57:57.782044 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:57.781995    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t2bhm" event={"ID":"3c7a3895-ea08-4b99-a07e-21e7fb8667a6","Type":"ContainerStarted","Data":"2ac40c0ccd2d14fbd309748554a0e8f44961d9e5427ef81e65656744970637ff"}
Sep 11 11:57:57.782420 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:57.782225    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t2bhm"
Sep 11 11:57:57.800479 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:57.800437    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t2bhm" podStartSLOduration=1.7255893100000002 podStartE2EDuration="6.800423084s" totalImagesPullingTime="5.074833774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:57:57.799185311 +0000 UTC m=+600.221053718" watchObservedRunningTime="2026-09-11 11:57:57.800423084 +0000 UTC m=+600.222291463"
Sep 11 11:57:58.054980 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:58.054904    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 11:57:58.057730 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:57:58.057704    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 11:58:03.815495 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:03.815468    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t2bhm"
Sep 11 11:58:04.344561 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.344518    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-zfn4h"]
Sep 11 11:58:04.350260 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.350238    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zfn4h"
Sep 11 11:58:04.352640 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.352619    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 11:58:04.353347 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.353328    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 11:58:04.353549 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.353335    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pq2dp\""
Sep 11 11:58:04.353981 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.353964    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-zfn4h"]
Sep 11 11:58:04.385905 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.385876    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb45r\" (UniqueName: \"kubernetes.io/projected/bdb25c15-618f-4790-9612-460f929e881a-kube-api-access-nb45r\") pod \"authorino-557f876f69-zfn4h\" (UID: \"bdb25c15-618f-4790-9612-460f929e881a\") " pod="kuadrant-system/authorino-557f876f69-zfn4h"
Sep 11 11:58:04.486332 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.486307    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb45r\" (UniqueName: \"kubernetes.io/projected/bdb25c15-618f-4790-9612-460f929e881a-kube-api-access-nb45r\") pod \"authorino-557f876f69-zfn4h\" (UID: \"bdb25c15-618f-4790-9612-460f929e881a\") " pod="kuadrant-system/authorino-557f876f69-zfn4h"
Sep 11 11:58:04.497072 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.497047    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb45r\" (UniqueName: \"kubernetes.io/projected/bdb25c15-618f-4790-9612-460f929e881a-kube-api-access-nb45r\") pod \"authorino-557f876f69-zfn4h\" (UID: \"bdb25c15-618f-4790-9612-460f929e881a\") " pod="kuadrant-system/authorino-557f876f69-zfn4h"
Sep 11 11:58:04.560193 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.560165    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zfn4h"]
Sep 11 11:58:04.561018 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.560795    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zfn4h"
Sep 11 11:58:04.583970 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.583940    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7df746fdd8-flfs4"]
Sep 11 11:58:04.588989 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.588968    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7df746fdd8-flfs4"
Sep 11 11:58:04.594138 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.594114    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7df746fdd8-flfs4"]
Sep 11 11:58:04.687432 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.687400    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2v52\" (UniqueName: \"kubernetes.io/projected/9a875750-5a28-43f0-bd65-d2734f84c7dc-kube-api-access-h2v52\") pod \"authorino-7df746fdd8-flfs4\" (UID: \"9a875750-5a28-43f0-bd65-d2734f84c7dc\") " pod="kuadrant-system/authorino-7df746fdd8-flfs4"
Sep 11 11:58:04.788424 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.788387    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2v52\" (UniqueName: \"kubernetes.io/projected/9a875750-5a28-43f0-bd65-d2734f84c7dc-kube-api-access-h2v52\") pod \"authorino-7df746fdd8-flfs4\" (UID: \"9a875750-5a28-43f0-bd65-d2734f84c7dc\") " pod="kuadrant-system/authorino-7df746fdd8-flfs4"
Sep 11 11:58:04.796798 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.796778    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2v52\" (UniqueName: \"kubernetes.io/projected/9a875750-5a28-43f0-bd65-d2734f84c7dc-kube-api-access-h2v52\") pod \"authorino-7df746fdd8-flfs4\" (UID: \"9a875750-5a28-43f0-bd65-d2734f84c7dc\") " pod="kuadrant-system/authorino-7df746fdd8-flfs4"
Sep 11 11:58:04.840916 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.840885    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zfn4h"]
Sep 11 11:58:04.841986 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:58:04.841957    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdb25c15_618f_4790_9612_460f929e881a.slice/crio-1799d69556539fc86d5183be8842367d25f04c42cbf9d39dcce03f6fd6ee88c0 WatchSource:0}: Error finding container 1799d69556539fc86d5183be8842367d25f04c42cbf9d39dcce03f6fd6ee88c0: Status 404 returned error can't find the container with id 1799d69556539fc86d5183be8842367d25f04c42cbf9d39dcce03f6fd6ee88c0
Sep 11 11:58:04.902088 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:04.902026    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7df746fdd8-flfs4"
Sep 11 11:58:05.173497 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:05.173475    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7df746fdd8-flfs4"]
Sep 11 11:58:05.175120 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:58:05.175099    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a875750_5a28_43f0_bd65_d2734f84c7dc.slice/crio-e537bc03e18252292400a6be98f5b4572c6183383732137de1c0d5616e48c902 WatchSource:0}: Error finding container e537bc03e18252292400a6be98f5b4572c6183383732137de1c0d5616e48c902: Status 404 returned error can't find the container with id e537bc03e18252292400a6be98f5b4572c6183383732137de1c0d5616e48c902
Sep 11 11:58:05.826945 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:05.826862    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7df746fdd8-flfs4" event={"ID":"9a875750-5a28-43f0-bd65-d2734f84c7dc","Type":"ContainerStarted","Data":"1171214040aca87ef48a58f9dba3ab6988871e04e75c7236103d0b021b5945d8"}
Sep 11 11:58:05.826945 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:05.826910    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7df746fdd8-flfs4" event={"ID":"9a875750-5a28-43f0-bd65-d2734f84c7dc","Type":"ContainerStarted","Data":"e537bc03e18252292400a6be98f5b4572c6183383732137de1c0d5616e48c902"}
Sep 11 11:58:05.828405 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:05.828380    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-zfn4h" event={"ID":"bdb25c15-618f-4790-9612-460f929e881a","Type":"ContainerStarted","Data":"d49dee86abe49213e30db36b7f850bfd96ad9afcc6d78e1730528e5c490a0a59"}
Sep 11 11:58:05.828519 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:05.828415    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-zfn4h" event={"ID":"bdb25c15-618f-4790-9612-460f929e881a","Type":"ContainerStarted","Data":"1799d69556539fc86d5183be8842367d25f04c42cbf9d39dcce03f6fd6ee88c0"}
Sep 11 11:58:05.828519 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:05.828448    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-zfn4h" podUID="bdb25c15-618f-4790-9612-460f929e881a" containerName="authorino" containerID="cri-o://d49dee86abe49213e30db36b7f850bfd96ad9afcc6d78e1730528e5c490a0a59" gracePeriod=30
Sep 11 11:58:05.844359 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:05.844318    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7df746fdd8-flfs4" podStartSLOduration=1.473977545 podStartE2EDuration="1.84430388s" totalImagesPullingTime="370.326335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:58:05.842303595 +0000 UTC m=+608.264171975" watchObservedRunningTime="2026-09-11 11:58:05.84430388 +0000 UTC m=+608.266172250"
Sep 11 11:58:05.859414 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:05.859372    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-zfn4h" podStartSLOduration=1.341516465 podStartE2EDuration="1.859357976s" totalImagesPullingTime="517.841511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:58:05.857817209 +0000 UTC m=+608.279685590" watchObservedRunningTime="2026-09-11 11:58:05.859357976 +0000 UTC m=+608.281226355"
Sep 11 11:58:05.961255 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:05.961223    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75788dd854-pfrfs"]
Sep 11 11:58:05.964559 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:05.964537    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75788dd854-pfrfs"
Sep 11 11:58:05.966817 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:05.966800    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 11:58:05.974568 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:05.974548    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75788dd854-pfrfs"]
Sep 11 11:58:05.998210 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:05.998183    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d41e2b3-72ba-40c9-b9c1-b9b439256b48-tls-cert\") pod \"authorino-75788dd854-pfrfs\" (UID: \"3d41e2b3-72ba-40c9-b9c1-b9b439256b48\") " pod="kuadrant-system/authorino-75788dd854-pfrfs"
Sep 11 11:58:05.998325 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:05.998296    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p5zk\" (UniqueName: \"kubernetes.io/projected/3d41e2b3-72ba-40c9-b9c1-b9b439256b48-kube-api-access-4p5zk\") pod \"authorino-75788dd854-pfrfs\" (UID: \"3d41e2b3-72ba-40c9-b9c1-b9b439256b48\") " pod="kuadrant-system/authorino-75788dd854-pfrfs"
Sep 11 11:58:06.070993 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.070973    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zfn4h"
Sep 11 11:58:06.099399 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.099336    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb45r\" (UniqueName: \"kubernetes.io/projected/bdb25c15-618f-4790-9612-460f929e881a-kube-api-access-nb45r\") pod \"bdb25c15-618f-4790-9612-460f929e881a\" (UID: \"bdb25c15-618f-4790-9612-460f929e881a\") "
Sep 11 11:58:06.099515 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.099475    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d41e2b3-72ba-40c9-b9c1-b9b439256b48-tls-cert\") pod \"authorino-75788dd854-pfrfs\" (UID: \"3d41e2b3-72ba-40c9-b9c1-b9b439256b48\") " pod="kuadrant-system/authorino-75788dd854-pfrfs"
Sep 11 11:58:06.099577 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.099526    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p5zk\" (UniqueName: \"kubernetes.io/projected/3d41e2b3-72ba-40c9-b9c1-b9b439256b48-kube-api-access-4p5zk\") pod \"authorino-75788dd854-pfrfs\" (UID: \"3d41e2b3-72ba-40c9-b9c1-b9b439256b48\") " pod="kuadrant-system/authorino-75788dd854-pfrfs"
Sep 11 11:58:06.101365 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.101337    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb25c15-618f-4790-9612-460f929e881a-kube-api-access-nb45r" (OuterVolumeSpecName: "kube-api-access-nb45r") pod "bdb25c15-618f-4790-9612-460f929e881a" (UID: "bdb25c15-618f-4790-9612-460f929e881a"). InnerVolumeSpecName "kube-api-access-nb45r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:58:06.101866 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.101847    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d41e2b3-72ba-40c9-b9c1-b9b439256b48-tls-cert\") pod \"authorino-75788dd854-pfrfs\" (UID: \"3d41e2b3-72ba-40c9-b9c1-b9b439256b48\") " pod="kuadrant-system/authorino-75788dd854-pfrfs"
Sep 11 11:58:06.107271 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.107242    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p5zk\" (UniqueName: \"kubernetes.io/projected/3d41e2b3-72ba-40c9-b9c1-b9b439256b48-kube-api-access-4p5zk\") pod \"authorino-75788dd854-pfrfs\" (UID: \"3d41e2b3-72ba-40c9-b9c1-b9b439256b48\") " pod="kuadrant-system/authorino-75788dd854-pfrfs"
Sep 11 11:58:06.200476 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.200453    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb45r\" (UniqueName: \"kubernetes.io/projected/bdb25c15-618f-4790-9612-460f929e881a-kube-api-access-nb45r\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:58:06.275240 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.275220    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75788dd854-pfrfs"
Sep 11 11:58:06.546342 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.546318    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75788dd854-pfrfs"]
Sep 11 11:58:06.547639 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:58:06.547613    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d41e2b3_72ba_40c9_b9c1_b9b439256b48.slice/crio-24b689f65a32e5817e9ea3069d51bdefd21a421f97603edbefd2a233772fc85f WatchSource:0}: Error finding container 24b689f65a32e5817e9ea3069d51bdefd21a421f97603edbefd2a233772fc85f: Status 404 returned error can't find the container with id 24b689f65a32e5817e9ea3069d51bdefd21a421f97603edbefd2a233772fc85f
Sep 11 11:58:06.839370 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.839345    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75788dd854-pfrfs" event={"ID":"3d41e2b3-72ba-40c9-b9c1-b9b439256b48","Type":"ContainerStarted","Data":"24b689f65a32e5817e9ea3069d51bdefd21a421f97603edbefd2a233772fc85f"}
Sep 11 11:58:06.840775 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.840744    2569 generic.go:356] "Generic (PLEG): container finished" podID="bdb25c15-618f-4790-9612-460f929e881a" containerID="d49dee86abe49213e30db36b7f850bfd96ad9afcc6d78e1730528e5c490a0a59" exitCode=0
Sep 11 11:58:06.840882 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.840786    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zfn4h"
Sep 11 11:58:06.840882 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.840804    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-zfn4h" event={"ID":"bdb25c15-618f-4790-9612-460f929e881a","Type":"ContainerDied","Data":"d49dee86abe49213e30db36b7f850bfd96ad9afcc6d78e1730528e5c490a0a59"}
Sep 11 11:58:06.840882 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.840854    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-zfn4h" event={"ID":"bdb25c15-618f-4790-9612-460f929e881a","Type":"ContainerDied","Data":"1799d69556539fc86d5183be8842367d25f04c42cbf9d39dcce03f6fd6ee88c0"}
Sep 11 11:58:06.840882 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.840880    2569 scope.go:117] "RemoveContainer" containerID="d49dee86abe49213e30db36b7f850bfd96ad9afcc6d78e1730528e5c490a0a59"
Sep 11 11:58:06.850728 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.850712    2569 scope.go:117] "RemoveContainer" containerID="d49dee86abe49213e30db36b7f850bfd96ad9afcc6d78e1730528e5c490a0a59"
Sep 11 11:58:06.851057 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:58:06.851035    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d49dee86abe49213e30db36b7f850bfd96ad9afcc6d78e1730528e5c490a0a59\": container with ID starting with d49dee86abe49213e30db36b7f850bfd96ad9afcc6d78e1730528e5c490a0a59 not found: ID does not exist" containerID="d49dee86abe49213e30db36b7f850bfd96ad9afcc6d78e1730528e5c490a0a59"
Sep 11 11:58:06.851110 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.851069    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49dee86abe49213e30db36b7f850bfd96ad9afcc6d78e1730528e5c490a0a59"} err="failed to get container status \"d49dee86abe49213e30db36b7f850bfd96ad9afcc6d78e1730528e5c490a0a59\": rpc error: code = NotFound desc = could not find container \"d49dee86abe49213e30db36b7f850bfd96ad9afcc6d78e1730528e5c490a0a59\": container with ID starting with d49dee86abe49213e30db36b7f850bfd96ad9afcc6d78e1730528e5c490a0a59 not found: ID does not exist"
Sep 11 11:58:06.863034 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.863008    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zfn4h"]
Sep 11 11:58:06.866607 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:06.866578    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-zfn4h"]
Sep 11 11:58:07.849030 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:07.848996    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75788dd854-pfrfs" event={"ID":"3d41e2b3-72ba-40c9-b9c1-b9b439256b48","Type":"ContainerStarted","Data":"71c0b80811160e2ad5a1f5bc333802b995b530585ea0a7836dc863305871774a"}
Sep 11 11:58:07.867114 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:07.867067    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75788dd854-pfrfs" podStartSLOduration=2.590290502 podStartE2EDuration="2.867052718s" totalImagesPullingTime="276.762216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:58:07.864936148 +0000 UTC m=+610.286804529" watchObservedRunningTime="2026-09-11 11:58:07.867052718 +0000 UTC m=+610.288921097"
Sep 11 11:58:07.907607 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:07.904520    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7df746fdd8-flfs4"]
Sep 11 11:58:07.908080 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:07.908044    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7df746fdd8-flfs4" podUID="9a875750-5a28-43f0-bd65-d2734f84c7dc" containerName="authorino" containerID="cri-o://1171214040aca87ef48a58f9dba3ab6988871e04e75c7236103d0b021b5945d8" gracePeriod=30
Sep 11 11:58:08.071334 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:08.071298    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdb25c15-618f-4790-9612-460f929e881a" path="/var/lib/kubelet/pods/bdb25c15-618f-4790-9612-460f929e881a/volumes"
Sep 11 11:58:08.214832 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:08.214809    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7df746fdd8-flfs4"
Sep 11 11:58:08.319682 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:08.319652    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2v52\" (UniqueName: \"kubernetes.io/projected/9a875750-5a28-43f0-bd65-d2734f84c7dc-kube-api-access-h2v52\") pod \"9a875750-5a28-43f0-bd65-d2734f84c7dc\" (UID: \"9a875750-5a28-43f0-bd65-d2734f84c7dc\") "
Sep 11 11:58:08.321695 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:08.321665    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a875750-5a28-43f0-bd65-d2734f84c7dc-kube-api-access-h2v52" (OuterVolumeSpecName: "kube-api-access-h2v52") pod "9a875750-5a28-43f0-bd65-d2734f84c7dc" (UID: "9a875750-5a28-43f0-bd65-d2734f84c7dc"). InnerVolumeSpecName "kube-api-access-h2v52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 11:58:08.420964 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:08.420892    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2v52\" (UniqueName: \"kubernetes.io/projected/9a875750-5a28-43f0-bd65-d2734f84c7dc-kube-api-access-h2v52\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 11:58:08.855607 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:08.855555    2569 generic.go:356] "Generic (PLEG): container finished" podID="9a875750-5a28-43f0-bd65-d2734f84c7dc" containerID="1171214040aca87ef48a58f9dba3ab6988871e04e75c7236103d0b021b5945d8" exitCode=0
Sep 11 11:58:08.855771 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:08.855610    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7df746fdd8-flfs4"
Sep 11 11:58:08.855771 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:08.855638    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7df746fdd8-flfs4" event={"ID":"9a875750-5a28-43f0-bd65-d2734f84c7dc","Type":"ContainerDied","Data":"1171214040aca87ef48a58f9dba3ab6988871e04e75c7236103d0b021b5945d8"}
Sep 11 11:58:08.855771 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:08.855680    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7df746fdd8-flfs4" event={"ID":"9a875750-5a28-43f0-bd65-d2734f84c7dc","Type":"ContainerDied","Data":"e537bc03e18252292400a6be98f5b4572c6183383732137de1c0d5616e48c902"}
Sep 11 11:58:08.855771 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:08.855694    2569 scope.go:117] "RemoveContainer" containerID="1171214040aca87ef48a58f9dba3ab6988871e04e75c7236103d0b021b5945d8"
Sep 11 11:58:08.864255 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:08.864240    2569 scope.go:117] "RemoveContainer" containerID="1171214040aca87ef48a58f9dba3ab6988871e04e75c7236103d0b021b5945d8"
Sep 11 11:58:08.864499 ip-10-0-143-220 kubenswrapper[2569]: E0911 11:58:08.864481    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1171214040aca87ef48a58f9dba3ab6988871e04e75c7236103d0b021b5945d8\": container with ID starting with 1171214040aca87ef48a58f9dba3ab6988871e04e75c7236103d0b021b5945d8 not found: ID does not exist" containerID="1171214040aca87ef48a58f9dba3ab6988871e04e75c7236103d0b021b5945d8"
Sep 11 11:58:08.864552 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:08.864506    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1171214040aca87ef48a58f9dba3ab6988871e04e75c7236103d0b021b5945d8"} err="failed to get container status \"1171214040aca87ef48a58f9dba3ab6988871e04e75c7236103d0b021b5945d8\": rpc error: code = NotFound desc = could not find container \"1171214040aca87ef48a58f9dba3ab6988871e04e75c7236103d0b021b5945d8\": container with ID starting with 1171214040aca87ef48a58f9dba3ab6988871e04e75c7236103d0b021b5945d8 not found: ID does not exist"
Sep 11 11:58:08.883193 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:08.883171    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7df746fdd8-flfs4"]
Sep 11 11:58:08.890298 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:08.890278    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7df746fdd8-flfs4"]
Sep 11 11:58:10.069162 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:10.069130    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a875750-5a28-43f0-bd65-d2734f84c7dc" path="/var/lib/kubelet/pods/9a875750-5a28-43f0-bd65-d2734f84c7dc/volumes"
Sep 11 11:58:10.915439 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:10.915411    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c4fd66f5c-w44qc"]
Sep 11 11:58:10.915804 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:10.915790    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a875750-5a28-43f0-bd65-d2734f84c7dc" containerName="authorino"
Sep 11 11:58:10.915804 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:10.915805    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a875750-5a28-43f0-bd65-d2734f84c7dc" containerName="authorino"
Sep 11 11:58:10.916135 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:10.915820    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb25c15-618f-4790-9612-460f929e881a" containerName="authorino"
Sep 11 11:58:10.916135 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:10.915826    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb25c15-618f-4790-9612-460f929e881a" containerName="authorino"
Sep 11 11:58:10.916135 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:10.915885    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb25c15-618f-4790-9612-460f929e881a" containerName="authorino"
Sep 11 11:58:10.916135 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:10.915893    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a875750-5a28-43f0-bd65-d2734f84c7dc" containerName="authorino"
Sep 11 11:58:10.925714 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:10.925697    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4fd66f5c-w44qc"
Sep 11 11:58:10.928881 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:10.928861    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 11:58:10.929768 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:10.929747    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qmtgj\""
Sep 11 11:58:10.930058 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:10.929972    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 11:58:10.930165 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:10.930086    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 11:58:10.933714 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:10.933363    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4fd66f5c-w44qc"]
Sep 11 11:58:11.044284 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:11.044262    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c72257ac-4829-4f36-8c0f-340082a19b2b-metrics-certs\") pod \"maas-controller-c4fd66f5c-w44qc\" (UID: \"c72257ac-4829-4f36-8c0f-340082a19b2b\") " pod="opendatahub/maas-controller-c4fd66f5c-w44qc"
Sep 11 11:58:11.044389 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:11.044319    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c72257ac-4829-4f36-8c0f-340082a19b2b-webhook-cert\") pod \"maas-controller-c4fd66f5c-w44qc\" (UID: \"c72257ac-4829-4f36-8c0f-340082a19b2b\") " pod="opendatahub/maas-controller-c4fd66f5c-w44qc"
Sep 11 11:58:11.044389 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:11.044337    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78lns\" (UniqueName: \"kubernetes.io/projected/c72257ac-4829-4f36-8c0f-340082a19b2b-kube-api-access-78lns\") pod \"maas-controller-c4fd66f5c-w44qc\" (UID: \"c72257ac-4829-4f36-8c0f-340082a19b2b\") " pod="opendatahub/maas-controller-c4fd66f5c-w44qc"
Sep 11 11:58:11.145686 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:11.145653    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c72257ac-4829-4f36-8c0f-340082a19b2b-webhook-cert\") pod \"maas-controller-c4fd66f5c-w44qc\" (UID: \"c72257ac-4829-4f36-8c0f-340082a19b2b\") " pod="opendatahub/maas-controller-c4fd66f5c-w44qc"
Sep 11 11:58:11.146003 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:11.145690    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78lns\" (UniqueName: \"kubernetes.io/projected/c72257ac-4829-4f36-8c0f-340082a19b2b-kube-api-access-78lns\") pod \"maas-controller-c4fd66f5c-w44qc\" (UID: \"c72257ac-4829-4f36-8c0f-340082a19b2b\") " pod="opendatahub/maas-controller-c4fd66f5c-w44qc"
Sep 11 11:58:11.146003 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:11.145731    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c72257ac-4829-4f36-8c0f-340082a19b2b-metrics-certs\") pod \"maas-controller-c4fd66f5c-w44qc\" (UID: \"c72257ac-4829-4f36-8c0f-340082a19b2b\") " pod="opendatahub/maas-controller-c4fd66f5c-w44qc"
Sep 11 11:58:11.148221 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:11.148196    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c72257ac-4829-4f36-8c0f-340082a19b2b-webhook-cert\") pod \"maas-controller-c4fd66f5c-w44qc\" (UID: \"c72257ac-4829-4f36-8c0f-340082a19b2b\") " pod="opendatahub/maas-controller-c4fd66f5c-w44qc"
Sep 11 11:58:11.148311 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:11.148280    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c72257ac-4829-4f36-8c0f-340082a19b2b-metrics-certs\") pod \"maas-controller-c4fd66f5c-w44qc\" (UID: \"c72257ac-4829-4f36-8c0f-340082a19b2b\") " pod="opendatahub/maas-controller-c4fd66f5c-w44qc"
Sep 11 11:58:11.154012 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:11.153993    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78lns\" (UniqueName: \"kubernetes.io/projected/c72257ac-4829-4f36-8c0f-340082a19b2b-kube-api-access-78lns\") pod \"maas-controller-c4fd66f5c-w44qc\" (UID: \"c72257ac-4829-4f36-8c0f-340082a19b2b\") " pod="opendatahub/maas-controller-c4fd66f5c-w44qc"
Sep 11 11:58:11.237190 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:11.237110    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4fd66f5c-w44qc"
Sep 11 11:58:11.512657 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:11.512631    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4fd66f5c-w44qc"]
Sep 11 11:58:11.514147 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:58:11.514120    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72257ac_4829_4f36_8c0f_340082a19b2b.slice/crio-9eea1e5e0fced4eb4e66edf7414d2a9535786ac5b43c73a049a96ebbb99ea783 WatchSource:0}: Error finding container 9eea1e5e0fced4eb4e66edf7414d2a9535786ac5b43c73a049a96ebbb99ea783: Status 404 returned error can't find the container with id 9eea1e5e0fced4eb4e66edf7414d2a9535786ac5b43c73a049a96ebbb99ea783
Sep 11 11:58:11.873007 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:11.872932    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4fd66f5c-w44qc" event={"ID":"c72257ac-4829-4f36-8c0f-340082a19b2b","Type":"ContainerStarted","Data":"9eea1e5e0fced4eb4e66edf7414d2a9535786ac5b43c73a049a96ebbb99ea783"}
Sep 11 11:58:13.886738 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:13.886706    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4fd66f5c-w44qc" event={"ID":"c72257ac-4829-4f36-8c0f-340082a19b2b","Type":"ContainerStarted","Data":"9f4397adbc7bc66d9be7d0a4a51d472b32c0f60fbfafdab235e9fcca1d099a15"}
Sep 11 11:58:13.887098 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:13.886816    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c4fd66f5c-w44qc"
Sep 11 11:58:13.909934 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:13.909881    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c4fd66f5c-w44qc" podStartSLOduration=1.699217714 podStartE2EDuration="3.909865511s" totalImagesPullingTime="2.210647797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:58:13.907393576 +0000 UTC m=+616.329261967" watchObservedRunningTime="2026-09-11 11:58:13.909865511 +0000 UTC m=+616.331733891"
Sep 11 11:58:24.894240 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:24.894166    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c4fd66f5c-w44qc"
Sep 11 11:58:49.870169 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:49.870137    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-5bqf9"]
Sep 11 11:58:49.876308 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:49.876291    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5bqf9"
Sep 11 11:58:49.879099 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:49.879071    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 11:58:49.879324 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:49.879296    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nsjb4\""
Sep 11 11:58:49.889162 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:49.889138    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-5bqf9"]
Sep 11 11:58:49.964039 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:49.964000    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkpn5\" (UniqueName: \"kubernetes.io/projected/c1595fd2-0403-487d-a7ea-3ae02b1309a7-kube-api-access-vkpn5\") pod \"payload-pre-processing-59fbc47bf8-5bqf9\" (UID: \"c1595fd2-0403-487d-a7ea-3ae02b1309a7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5bqf9"
Sep 11 11:58:49.964039 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:49.964046    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1595fd2-0403-487d-a7ea-3ae02b1309a7-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-5bqf9\" (UID: \"c1595fd2-0403-487d-a7ea-3ae02b1309a7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5bqf9"
Sep 11 11:58:50.065552 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:50.065518    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkpn5\" (UniqueName: \"kubernetes.io/projected/c1595fd2-0403-487d-a7ea-3ae02b1309a7-kube-api-access-vkpn5\") pod \"payload-pre-processing-59fbc47bf8-5bqf9\" (UID: \"c1595fd2-0403-487d-a7ea-3ae02b1309a7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5bqf9"
Sep 11 11:58:50.065765 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:50.065579    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1595fd2-0403-487d-a7ea-3ae02b1309a7-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-5bqf9\" (UID: \"c1595fd2-0403-487d-a7ea-3ae02b1309a7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5bqf9"
Sep 11 11:58:50.066307 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:50.066280    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1595fd2-0403-487d-a7ea-3ae02b1309a7-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-5bqf9\" (UID: \"c1595fd2-0403-487d-a7ea-3ae02b1309a7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5bqf9"
Sep 11 11:58:50.074776 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:50.074744    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkpn5\" (UniqueName: \"kubernetes.io/projected/c1595fd2-0403-487d-a7ea-3ae02b1309a7-kube-api-access-vkpn5\") pod \"payload-pre-processing-59fbc47bf8-5bqf9\" (UID: \"c1595fd2-0403-487d-a7ea-3ae02b1309a7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5bqf9"
Sep 11 11:58:50.191171 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:50.191084    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5bqf9"
Sep 11 11:58:50.468792 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:50.468770    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-5bqf9"]
Sep 11 11:58:50.471657 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:58:50.471627    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1595fd2_0403_487d_a7ea_3ae02b1309a7.slice/crio-5e9afe513220007f2c04fd9e6df87b2cdc0915acb3afa3cebb9353e33c8af75a WatchSource:0}: Error finding container 5e9afe513220007f2c04fd9e6df87b2cdc0915acb3afa3cebb9353e33c8af75a: Status 404 returned error can't find the container with id 5e9afe513220007f2c04fd9e6df87b2cdc0915acb3afa3cebb9353e33c8af75a
Sep 11 11:58:50.473496 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:50.473476    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 11:58:51.092710 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:51.092680    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5bqf9" event={"ID":"c1595fd2-0403-487d-a7ea-3ae02b1309a7","Type":"ContainerStarted","Data":"5e9afe513220007f2c04fd9e6df87b2cdc0915acb3afa3cebb9353e33c8af75a"}
Sep 11 11:58:53.105269 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:53.105234    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5bqf9" event={"ID":"c1595fd2-0403-487d-a7ea-3ae02b1309a7","Type":"ContainerStarted","Data":"af613c9a463ba086644c27ae33aee6df8af6ef1b872fef56705c1d88479a8ea0"}
Sep 11 11:58:53.105640 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:53.105352    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5bqf9"
Sep 11 11:58:53.126495 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:58:53.126447    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5bqf9" podStartSLOduration=2.420706091 podStartE2EDuration="4.126429693s" totalImagesPullingTime="1.705723602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:58:53.126074203 +0000 UTC m=+655.547942594" watchObservedRunningTime="2026-09-11 11:58:53.126429693 +0000 UTC m=+655.548298076"
Sep 11 11:59:04.112685 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:04.112650    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5bqf9"
Sep 11 11:59:09.754737 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.754704    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"]
Sep 11 11:59:09.759477 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.759458    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.762903 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.762880    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 11:59:09.763031 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.762918    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 11:59:09.763031 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.762880    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fg5rd\""
Sep 11 11:59:09.763031 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.762923    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 11 11:59:09.767319 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.767296    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"]
Sep 11 11:59:09.822751 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.822728    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.822861 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.822763    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.822861 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.822792    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86xnw\" (UniqueName: \"kubernetes.io/projected/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-kube-api-access-86xnw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.822861 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.822848    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.822983 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.822869    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.822983 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.822886    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.923490 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.923462    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.923604 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.923502    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.923658 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.923619    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86xnw\" (UniqueName: \"kubernetes.io/projected/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-kube-api-access-86xnw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.923711 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.923665    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.923711 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.923693    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.923811 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.923718    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.923861 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.923841    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.924012 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.923991    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.924078 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.924019    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.925838 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.925812    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.926036 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.926021    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:09.933764 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:09.933740    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86xnw\" (UniqueName: \"kubernetes.io/projected/03c3f321-c61d-47e5-8a7a-0ead31f5e6dd-kube-api-access-86xnw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh\" (UID: \"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:10.070640 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:10.070552    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:10.347652 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:10.347618    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"]
Sep 11 11:59:10.351016 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:59:10.349980    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c3f321_c61d_47e5_8a7a_0ead31f5e6dd.slice/crio-dd7f32bfd9dbe62571bc8c171a1fde43f27036b734e150390d99154c99552d2c WatchSource:0}: Error finding container dd7f32bfd9dbe62571bc8c171a1fde43f27036b734e150390d99154c99552d2c: Status 404 returned error can't find the container with id dd7f32bfd9dbe62571bc8c171a1fde43f27036b734e150390d99154c99552d2c
Sep 11 11:59:11.206381 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:11.206341    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh" event={"ID":"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd","Type":"ContainerStarted","Data":"dd7f32bfd9dbe62571bc8c171a1fde43f27036b734e150390d99154c99552d2c"}
Sep 11 11:59:12.213223 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:12.213186    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh" event={"ID":"03c3f321-c61d-47e5-8a7a-0ead31f5e6dd","Type":"ContainerStarted","Data":"6309f0d49fa29493cf8f20afa11609ed59937d0c60c3935c0447f9e4aa6ae3a9"}
Sep 11 11:59:12.234951 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:12.234898    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh" podStartSLOduration=2.206078817 podStartE2EDuration="3.234882033s" totalImagesPullingTime="1.028803216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:59:12.232522503 +0000 UTC m=+674.654390884" watchObservedRunningTime="2026-09-11 11:59:12.234882033 +0000 UTC m=+674.656750426"
Sep 11 11:59:20.071130 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:20.071097    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:20.071499 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:20.071151    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:20.083715 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:20.083693    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:20.275747 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:20.275716    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh"
Sep 11 11:59:30.837781 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:30.837744    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"]
Sep 11 11:59:30.840186 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:30.840168    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:30.842938 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:30.842921    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 11 11:59:30.858562 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:30.858541    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"]
Sep 11 11:59:31.001134 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.001100    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/320bb8fb-1a01-4e74-804d-b53916b0458e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.001294 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.001150    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h65t\" (UniqueName: \"kubernetes.io/projected/320bb8fb-1a01-4e74-804d-b53916b0458e-kube-api-access-2h65t\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.001294 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.001205    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/320bb8fb-1a01-4e74-804d-b53916b0458e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.001294 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.001250    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/320bb8fb-1a01-4e74-804d-b53916b0458e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.001294 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.001292    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/320bb8fb-1a01-4e74-804d-b53916b0458e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.001443 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.001385    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/320bb8fb-1a01-4e74-804d-b53916b0458e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.102760 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.102677    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/320bb8fb-1a01-4e74-804d-b53916b0458e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.102760 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.102715    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/320bb8fb-1a01-4e74-804d-b53916b0458e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.102760 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.102750    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h65t\" (UniqueName: \"kubernetes.io/projected/320bb8fb-1a01-4e74-804d-b53916b0458e-kube-api-access-2h65t\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.103031 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.102778    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/320bb8fb-1a01-4e74-804d-b53916b0458e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.103031 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.102811    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/320bb8fb-1a01-4e74-804d-b53916b0458e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.103031 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.102848    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/320bb8fb-1a01-4e74-804d-b53916b0458e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.103184 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.103126    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/320bb8fb-1a01-4e74-804d-b53916b0458e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.103227 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.103194    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/320bb8fb-1a01-4e74-804d-b53916b0458e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.103227 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.103207    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/320bb8fb-1a01-4e74-804d-b53916b0458e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.105051 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.105027    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/320bb8fb-1a01-4e74-804d-b53916b0458e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.105291 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.105276    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/320bb8fb-1a01-4e74-804d-b53916b0458e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.126867 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.126843    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h65t\" (UniqueName: \"kubernetes.io/projected/320bb8fb-1a01-4e74-804d-b53916b0458e-kube-api-access-2h65t\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cj6n2\" (UID: \"320bb8fb-1a01-4e74-804d-b53916b0458e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.149771 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.149749    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:31.425541 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:31.425516    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"]
Sep 11 11:59:31.426061 ip-10-0-143-220 kubenswrapper[2569]: W0911 11:59:31.426036    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320bb8fb_1a01_4e74_804d_b53916b0458e.slice/crio-947cace75962ed8302fa3fec0ba37b217e987426e0f7bb6d969670aab503cb10 WatchSource:0}: Error finding container 947cace75962ed8302fa3fec0ba37b217e987426e0f7bb6d969670aab503cb10: Status 404 returned error can't find the container with id 947cace75962ed8302fa3fec0ba37b217e987426e0f7bb6d969670aab503cb10
Sep 11 11:59:32.327449 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:32.327401    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2" event={"ID":"320bb8fb-1a01-4e74-804d-b53916b0458e","Type":"ContainerStarted","Data":"01a0d24e145083e6926735300dec427348231d2822cf800bea938c900b230a11"}
Sep 11 11:59:32.327892 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:32.327477    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2" event={"ID":"320bb8fb-1a01-4e74-804d-b53916b0458e","Type":"ContainerStarted","Data":"947cace75962ed8302fa3fec0ba37b217e987426e0f7bb6d969670aab503cb10"}
Sep 11 11:59:32.350776 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:32.350726    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2" podStartSLOduration=2.160458173 podStartE2EDuration="2.350709587s" totalImagesPullingTime="190.251414ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:59:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:59:32.348326767 +0000 UTC m=+694.770195160" watchObservedRunningTime="2026-09-11 11:59:32.350709587 +0000 UTC m=+694.772577966"
Sep 11 11:59:41.150347 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:41.150313    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:41.150801 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:41.150370    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:41.162723 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:41.162700    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 11:59:41.387967 ip-10-0-143-220 kubenswrapper[2569]: I0911 11:59:41.387942    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cj6n2"
Sep 11 12:00:48.141222 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.141186    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"]
Sep 11 12:00:48.145283 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.145256    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.147812 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.147790    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 12:00:48.155965 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.155941    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"]
Sep 11 12:00:48.334104 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.334063    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43ba295e-5044-4635-8a7d-0550dae17127-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.334273 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.334110    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ba295e-5044-4635-8a7d-0550dae17127-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.334273 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.334150    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnqj9\" (UniqueName: \"kubernetes.io/projected/43ba295e-5044-4635-8a7d-0550dae17127-kube-api-access-mnqj9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.334273 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.334255    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ba295e-5044-4635-8a7d-0550dae17127-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.334446 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.334294    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ba295e-5044-4635-8a7d-0550dae17127-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.334446 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.334313    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ba295e-5044-4635-8a7d-0550dae17127-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.435956 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.435865    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ba295e-5044-4635-8a7d-0550dae17127-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.435956 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.435912    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ba295e-5044-4635-8a7d-0550dae17127-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.435956 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.435931    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ba295e-5044-4635-8a7d-0550dae17127-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.436214 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.436037    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43ba295e-5044-4635-8a7d-0550dae17127-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.436214 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.436062    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ba295e-5044-4635-8a7d-0550dae17127-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.436214 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.436090    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnqj9\" (UniqueName: \"kubernetes.io/projected/43ba295e-5044-4635-8a7d-0550dae17127-kube-api-access-mnqj9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.436389 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.436365    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ba295e-5044-4635-8a7d-0550dae17127-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.436451 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.436412    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43ba295e-5044-4635-8a7d-0550dae17127-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.436505 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.436442    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ba295e-5044-4635-8a7d-0550dae17127-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.438273 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.438242    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ba295e-5044-4635-8a7d-0550dae17127-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.438478 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.438462    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ba295e-5044-4635-8a7d-0550dae17127-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.445454 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.445433    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnqj9\" (UniqueName: \"kubernetes.io/projected/43ba295e-5044-4635-8a7d-0550dae17127-kube-api-access-mnqj9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc\" (UID: \"43ba295e-5044-4635-8a7d-0550dae17127\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.455301 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.455281    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:48.735618 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.735572    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"]
Sep 11 12:00:48.738479 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:00:48.738454    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43ba295e_5044_4635_8a7d_0550dae17127.slice/crio-e33ffb1d173e4e77024f90de52f1253a16e538ae9389b2e4f13b1bae4cc11508 WatchSource:0}: Error finding container e33ffb1d173e4e77024f90de52f1253a16e538ae9389b2e4f13b1bae4cc11508: Status 404 returned error can't find the container with id e33ffb1d173e4e77024f90de52f1253a16e538ae9389b2e4f13b1bae4cc11508
Sep 11 12:00:48.749749 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:48.749723    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc" event={"ID":"43ba295e-5044-4635-8a7d-0550dae17127","Type":"ContainerStarted","Data":"e33ffb1d173e4e77024f90de52f1253a16e538ae9389b2e4f13b1bae4cc11508"}
Sep 11 12:00:49.757284 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:49.757242    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc" event={"ID":"43ba295e-5044-4635-8a7d-0550dae17127","Type":"ContainerStarted","Data":"df73fc15e95dd22d45922e5a996f62a8b894e8e5e829e29d27574c7425343de9"}
Sep 11 12:00:49.781049 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:49.780992    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc" podStartSLOduration=1.533761884 podStartE2EDuration="1.780973906s" totalImagesPullingTime="247.212022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:00:49.778993307 +0000 UTC m=+772.200861687" watchObservedRunningTime="2026-09-11 12:00:49.780973906 +0000 UTC m=+772.202842308"
Sep 11 12:00:58.456387 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:58.456352    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:58.456862 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:58.456400    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:58.468853 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:58.468830    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:00:58.819562 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:00:58.819534    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc"
Sep 11 12:01:49.118563 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:49.118532    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55b4c84b95-wvdvm"]
Sep 11 12:01:49.122314 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:49.122289    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55b4c84b95-wvdvm"
Sep 11 12:01:49.133152 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:49.133127    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55b4c84b95-wvdvm"]
Sep 11 12:01:49.153983 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:49.153956    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/712d2398-1e74-4859-9466-587d71ae859c-tls-cert\") pod \"authorino-55b4c84b95-wvdvm\" (UID: \"712d2398-1e74-4859-9466-587d71ae859c\") " pod="kuadrant-system/authorino-55b4c84b95-wvdvm"
Sep 11 12:01:49.154118 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:49.154070    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5gd9\" (UniqueName: \"kubernetes.io/projected/712d2398-1e74-4859-9466-587d71ae859c-kube-api-access-x5gd9\") pod \"authorino-55b4c84b95-wvdvm\" (UID: \"712d2398-1e74-4859-9466-587d71ae859c\") " pod="kuadrant-system/authorino-55b4c84b95-wvdvm"
Sep 11 12:01:49.255312 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:49.255277    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5gd9\" (UniqueName: \"kubernetes.io/projected/712d2398-1e74-4859-9466-587d71ae859c-kube-api-access-x5gd9\") pod \"authorino-55b4c84b95-wvdvm\" (UID: \"712d2398-1e74-4859-9466-587d71ae859c\") " pod="kuadrant-system/authorino-55b4c84b95-wvdvm"
Sep 11 12:01:49.255472 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:49.255374    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/712d2398-1e74-4859-9466-587d71ae859c-tls-cert\") pod \"authorino-55b4c84b95-wvdvm\" (UID: \"712d2398-1e74-4859-9466-587d71ae859c\") " pod="kuadrant-system/authorino-55b4c84b95-wvdvm"
Sep 11 12:01:49.257797 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:49.257774    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/712d2398-1e74-4859-9466-587d71ae859c-tls-cert\") pod \"authorino-55b4c84b95-wvdvm\" (UID: \"712d2398-1e74-4859-9466-587d71ae859c\") " pod="kuadrant-system/authorino-55b4c84b95-wvdvm"
Sep 11 12:01:49.264135 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:49.264115    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5gd9\" (UniqueName: \"kubernetes.io/projected/712d2398-1e74-4859-9466-587d71ae859c-kube-api-access-x5gd9\") pod \"authorino-55b4c84b95-wvdvm\" (UID: \"712d2398-1e74-4859-9466-587d71ae859c\") " pod="kuadrant-system/authorino-55b4c84b95-wvdvm"
Sep 11 12:01:49.431255 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:49.431200    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55b4c84b95-wvdvm"
Sep 11 12:01:49.708673 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:49.708650    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55b4c84b95-wvdvm"]
Sep 11 12:01:49.710360 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:01:49.710331    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod712d2398_1e74_4859_9466_587d71ae859c.slice/crio-a2280ebe8d1212584a0ecf12681983942acd4043c8d3876b76041e7f001bb8e8 WatchSource:0}: Error finding container a2280ebe8d1212584a0ecf12681983942acd4043c8d3876b76041e7f001bb8e8: Status 404 returned error can't find the container with id a2280ebe8d1212584a0ecf12681983942acd4043c8d3876b76041e7f001bb8e8
Sep 11 12:01:50.098295 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:50.098261    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55b4c84b95-wvdvm" event={"ID":"712d2398-1e74-4859-9466-587d71ae859c","Type":"ContainerStarted","Data":"a2280ebe8d1212584a0ecf12681983942acd4043c8d3876b76041e7f001bb8e8"}
Sep 11 12:01:51.106170 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:51.106064    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55b4c84b95-wvdvm" event={"ID":"712d2398-1e74-4859-9466-587d71ae859c","Type":"ContainerStarted","Data":"1ce6a17c45a0ac25de1cae4bec8a5fa236f8a393b6a5795881436cd43c53192e"}
Sep 11 12:01:51.162611 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:51.161174    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55b4c84b95-wvdvm" podStartSLOduration=1.019908325 podStartE2EDuration="2.161147494s" totalImagesPullingTime="1.141239169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:01:51.124075826 +0000 UTC m=+833.545944205" watchObservedRunningTime="2026-09-11 12:01:51.161147494 +0000 UTC m=+833.583015874"
Sep 11 12:01:51.163747 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:51.163718    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75788dd854-pfrfs"]
Sep 11 12:01:51.164089 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:51.163914    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75788dd854-pfrfs" podUID="3d41e2b3-72ba-40c9-b9c1-b9b439256b48" containerName="authorino" containerID="cri-o://71c0b80811160e2ad5a1f5bc333802b995b530585ea0a7836dc863305871774a" gracePeriod=30
Sep 11 12:01:51.410810 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:51.410787    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75788dd854-pfrfs"
Sep 11 12:01:51.477640 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:51.477580    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4p5zk\" (UniqueName: \"kubernetes.io/projected/3d41e2b3-72ba-40c9-b9c1-b9b439256b48-kube-api-access-4p5zk\") pod \"3d41e2b3-72ba-40c9-b9c1-b9b439256b48\" (UID: \"3d41e2b3-72ba-40c9-b9c1-b9b439256b48\") "
Sep 11 12:01:51.477827 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:51.477703    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d41e2b3-72ba-40c9-b9c1-b9b439256b48-tls-cert\") pod \"3d41e2b3-72ba-40c9-b9c1-b9b439256b48\" (UID: \"3d41e2b3-72ba-40c9-b9c1-b9b439256b48\") "
Sep 11 12:01:51.479723 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:51.479690    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d41e2b3-72ba-40c9-b9c1-b9b439256b48-kube-api-access-4p5zk" (OuterVolumeSpecName: "kube-api-access-4p5zk") pod "3d41e2b3-72ba-40c9-b9c1-b9b439256b48" (UID: "3d41e2b3-72ba-40c9-b9c1-b9b439256b48"). InnerVolumeSpecName "kube-api-access-4p5zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:01:51.488900 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:51.488874    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d41e2b3-72ba-40c9-b9c1-b9b439256b48-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3d41e2b3-72ba-40c9-b9c1-b9b439256b48" (UID: "3d41e2b3-72ba-40c9-b9c1-b9b439256b48"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:01:51.579031 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:51.578996    2569 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d41e2b3-72ba-40c9-b9c1-b9b439256b48-tls-cert\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:01:51.579031 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:51.579022    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p5zk\" (UniqueName: \"kubernetes.io/projected/3d41e2b3-72ba-40c9-b9c1-b9b439256b48-kube-api-access-4p5zk\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:01:52.113494 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:52.113461    2569 generic.go:356] "Generic (PLEG): container finished" podID="3d41e2b3-72ba-40c9-b9c1-b9b439256b48" containerID="71c0b80811160e2ad5a1f5bc333802b995b530585ea0a7836dc863305871774a" exitCode=0
Sep 11 12:01:52.113953 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:52.113510    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75788dd854-pfrfs" event={"ID":"3d41e2b3-72ba-40c9-b9c1-b9b439256b48","Type":"ContainerDied","Data":"71c0b80811160e2ad5a1f5bc333802b995b530585ea0a7836dc863305871774a"}
Sep 11 12:01:52.113953 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:52.113553    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75788dd854-pfrfs" event={"ID":"3d41e2b3-72ba-40c9-b9c1-b9b439256b48","Type":"ContainerDied","Data":"24b689f65a32e5817e9ea3069d51bdefd21a421f97603edbefd2a233772fc85f"}
Sep 11 12:01:52.113953 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:52.113571    2569 scope.go:117] "RemoveContainer" containerID="71c0b80811160e2ad5a1f5bc333802b995b530585ea0a7836dc863305871774a"
Sep 11 12:01:52.113953 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:52.113519    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75788dd854-pfrfs"
Sep 11 12:01:52.122896 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:52.122881    2569 scope.go:117] "RemoveContainer" containerID="71c0b80811160e2ad5a1f5bc333802b995b530585ea0a7836dc863305871774a"
Sep 11 12:01:52.123235 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:01:52.123214    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c0b80811160e2ad5a1f5bc333802b995b530585ea0a7836dc863305871774a\": container with ID starting with 71c0b80811160e2ad5a1f5bc333802b995b530585ea0a7836dc863305871774a not found: ID does not exist" containerID="71c0b80811160e2ad5a1f5bc333802b995b530585ea0a7836dc863305871774a"
Sep 11 12:01:52.123326 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:52.123244    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c0b80811160e2ad5a1f5bc333802b995b530585ea0a7836dc863305871774a"} err="failed to get container status \"71c0b80811160e2ad5a1f5bc333802b995b530585ea0a7836dc863305871774a\": rpc error: code = NotFound desc = could not find container \"71c0b80811160e2ad5a1f5bc333802b995b530585ea0a7836dc863305871774a\": container with ID starting with 71c0b80811160e2ad5a1f5bc333802b995b530585ea0a7836dc863305871774a not found: ID does not exist"
Sep 11 12:01:52.145119 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:52.145091    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75788dd854-pfrfs"]
Sep 11 12:01:52.153833 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:52.153813    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75788dd854-pfrfs"]
Sep 11 12:01:54.070009 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:01:54.069977    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d41e2b3-72ba-40c9-b9c1-b9b439256b48" path="/var/lib/kubelet/pods/3d41e2b3-72ba-40c9-b9c1-b9b439256b48/volumes"
Sep 11 12:02:10.259947 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:10.259915    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7474-249f61"]
Sep 11 12:02:10.260405 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:10.260289    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d41e2b3-72ba-40c9-b9c1-b9b439256b48" containerName="authorino"
Sep 11 12:02:10.260405 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:10.260299    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d41e2b3-72ba-40c9-b9c1-b9b439256b48" containerName="authorino"
Sep 11 12:02:10.260405 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:10.260380    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d41e2b3-72ba-40c9-b9c1-b9b439256b48" containerName="authorino"
Sep 11 12:02:10.264684 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:10.264665    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7474-249f61"
Sep 11 12:02:10.267441 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:10.267416    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wkgmv\""
Sep 11 12:02:10.273028 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:10.272910    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7474-249f61"]
Sep 11 12:02:10.334240 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:10.334214    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6m58\" (UniqueName: \"kubernetes.io/projected/b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a-kube-api-access-n6m58\") pod \"test-curl-7474-249f61\" (UID: \"b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a\") " pod="opendatahub/test-curl-7474-249f61"
Sep 11 12:02:10.435474 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:10.435447    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6m58\" (UniqueName: \"kubernetes.io/projected/b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a-kube-api-access-n6m58\") pod \"test-curl-7474-249f61\" (UID: \"b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a\") " pod="opendatahub/test-curl-7474-249f61"
Sep 11 12:02:10.443701 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:10.443682    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6m58\" (UniqueName: \"kubernetes.io/projected/b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a-kube-api-access-n6m58\") pod \"test-curl-7474-249f61\" (UID: \"b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a\") " pod="opendatahub/test-curl-7474-249f61"
Sep 11 12:02:10.575683 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:10.575580    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7474-249f61"
Sep 11 12:02:10.848905 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:10.848879    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7474-249f61"]
Sep 11 12:02:10.850484 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:02:10.850458    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9e6fa6f_87c4_42f8_9ab5_f1284fea7a1a.slice/crio-2822532a8cbb9188b49b0732b0104945bb5c7e18c80ca508ce9ec40a33fa181f WatchSource:0}: Error finding container 2822532a8cbb9188b49b0732b0104945bb5c7e18c80ca508ce9ec40a33fa181f: Status 404 returned error can't find the container with id 2822532a8cbb9188b49b0732b0104945bb5c7e18c80ca508ce9ec40a33fa181f
Sep 11 12:02:11.216877 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:11.216800    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7474-249f61" event={"ID":"b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a","Type":"ContainerStarted","Data":"2822532a8cbb9188b49b0732b0104945bb5c7e18c80ca508ce9ec40a33fa181f"}
Sep 11 12:02:14.248642 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:14.248599    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7474-249f61" event={"ID":"b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a","Type":"ContainerStarted","Data":"cc705769280f93aad5b99ae6814a499f7fa5799eeb3c2b02e403e989257af9df"}
Sep 11 12:02:15.040040 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:15.039987    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7474-249f61" podStartSLOduration=2.461657994 podStartE2EDuration="5.039968956s" totalImagesPullingTime="2.578310962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:02:14.272232537 +0000 UTC m=+856.694100918" watchObservedRunningTime="2026-09-11 12:02:15.039968956 +0000 UTC m=+857.461837371"
Sep 11 12:02:15.041636 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:15.041615    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7474-249f61"]
Sep 11 12:02:15.048794 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:15.048762    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7474-249f61"]
Sep 11 12:02:15.164231 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:15.164155    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7474-b6e1fb"]
Sep 11 12:02:15.164619 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:15.164573    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a" containerName="test-curl-7474-249f61"
Sep 11 12:02:15.164619 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:15.164609    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a" containerName="test-curl-7474-249f61"
Sep 11 12:02:15.164765 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:15.164745    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a" containerName="test-curl-7474-249f61"
Sep 11 12:02:15.167836 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:15.167818    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7474-b6e1fb"
Sep 11 12:02:15.181205 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:15.181183    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7474-b6e1fb"]
Sep 11 12:02:15.254724 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:02:15.254688    2569 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7474-249f61" container="test-curl-7474-249f61"
Sep 11 12:02:15.280787 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:15.280761    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54v7d\" (UniqueName: \"kubernetes.io/projected/d055e321-1bc4-42cf-b336-9e78ea57b484-kube-api-access-54v7d\") pod \"test-curl-7474-b6e1fb\" (UID: \"d055e321-1bc4-42cf-b336-9e78ea57b484\") " pod="opendatahub/test-curl-7474-b6e1fb"
Sep 11 12:02:15.382087 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:15.382050    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54v7d\" (UniqueName: \"kubernetes.io/projected/d055e321-1bc4-42cf-b336-9e78ea57b484-kube-api-access-54v7d\") pod \"test-curl-7474-b6e1fb\" (UID: \"d055e321-1bc4-42cf-b336-9e78ea57b484\") " pod="opendatahub/test-curl-7474-b6e1fb"
Sep 11 12:02:15.390699 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:15.390673    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54v7d\" (UniqueName: \"kubernetes.io/projected/d055e321-1bc4-42cf-b336-9e78ea57b484-kube-api-access-54v7d\") pod \"test-curl-7474-b6e1fb\" (UID: \"d055e321-1bc4-42cf-b336-9e78ea57b484\") " pod="opendatahub/test-curl-7474-b6e1fb"
Sep 11 12:02:15.476490 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:15.476428    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7474-b6e1fb"
Sep 11 12:02:15.755405 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:15.755375    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7474-b6e1fb"]
Sep 11 12:02:15.755982 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:02:15.755957    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd055e321_1bc4_42cf_b336_9e78ea57b484.slice/crio-ea940723828c179919b5c7f05a3f39133598161d4d44f189e362d8664f848294 WatchSource:0}: Error finding container ea940723828c179919b5c7f05a3f39133598161d4d44f189e362d8664f848294: Status 404 returned error can't find the container with id ea940723828c179919b5c7f05a3f39133598161d4d44f189e362d8664f848294
Sep 11 12:02:16.261555 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:16.261524    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7474-b6e1fb" event={"ID":"d055e321-1bc4-42cf-b336-9e78ea57b484","Type":"ContainerStarted","Data":"23ebeece1dc93bd1786d7643b0c0f3a48b58a23295ba0e5f58f182351d480a1b"}
Sep 11 12:02:16.262034 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:16.261564    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7474-b6e1fb" event={"ID":"d055e321-1bc4-42cf-b336-9e78ea57b484","Type":"ContainerStarted","Data":"ea940723828c179919b5c7f05a3f39133598161d4d44f189e362d8664f848294"}
Sep 11 12:02:16.262034 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:16.261665    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7474-249f61" podUID="b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a" containerName="test-curl-7474-249f61" containerID="cri-o://cc705769280f93aad5b99ae6814a499f7fa5799eeb3c2b02e403e989257af9df" gracePeriod=2
Sep 11 12:02:16.290190 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:16.290149    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7474-b6e1fb" podStartSLOduration=1.290135535 podStartE2EDuration="1.290135535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:02:16.288726693 +0000 UTC m=+858.710595074" watchObservedRunningTime="2026-09-11 12:02:16.290135535 +0000 UTC m=+858.712003914"
Sep 11 12:02:16.971967 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:16.971934    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7474-b6e1fb"]
Sep 11 12:02:16.978098 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:16.978070    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7474-b6e1fb"]
Sep 11 12:02:17.266417 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:02:17.266388    2569 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7474-b6e1fb" container="test-curl-7474-b6e1fb"
Sep 11 12:02:18.274189 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.274136    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7474-b6e1fb" podUID="d055e321-1bc4-42cf-b336-9e78ea57b484" containerName="test-curl-7474-b6e1fb" containerID="cri-o://23ebeece1dc93bd1786d7643b0c0f3a48b58a23295ba0e5f58f182351d480a1b" gracePeriod=2
Sep 11 12:02:18.301034 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.301010    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"]
Sep 11 12:02:18.301553 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.301531    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d055e321-1bc4-42cf-b336-9e78ea57b484" containerName="test-curl-7474-b6e1fb"
Sep 11 12:02:18.301553 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.301552    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d055e321-1bc4-42cf-b336-9e78ea57b484" containerName="test-curl-7474-b6e1fb"
Sep 11 12:02:18.301744 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.301659    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d055e321-1bc4-42cf-b336-9e78ea57b484" containerName="test-curl-7474-b6e1fb"
Sep 11 12:02:18.305165 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.305135    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.310676 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.310656    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 12:02:18.311945 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.311924    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-49887c13-openshift-default-dockercfg-j6z9j\""
Sep 11 12:02:18.408615 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.408295    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-workload-socket\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.408615 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.408356    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38a7516-2ebf-4d8a-94af-5428d538e584-istiod-ca-cert\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.409206 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.409175    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-data\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.409315 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.409216    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-envoy\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.409315 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.409281    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z6fw\" (UniqueName: \"kubernetes.io/projected/b38a7516-2ebf-4d8a-94af-5428d538e584-kube-api-access-9z6fw\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.409495 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.409363    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-workload-certs\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.409495 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.409390    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-podinfo\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.409495 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.409418    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-credential-socket\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.409495 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.409452    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-token\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.421607 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.420405    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"]
Sep 11 12:02:18.464852 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.464829    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7474-249f61"
Sep 11 12:02:18.510520 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.510494    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-workload-certs\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.510677 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.510525    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-podinfo\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.510677 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.510546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-credential-socket\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.510677 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.510564    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-token\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.510810 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.510682    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-workload-socket\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.510810 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.510705    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38a7516-2ebf-4d8a-94af-5428d538e584-istiod-ca-cert\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.510950 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.510925    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-workload-certs\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.511049 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.511028    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-data\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.511049 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.511038    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-workload-socket\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.511193 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.511078    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-envoy\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.511193 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.511145    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z6fw\" (UniqueName: \"kubernetes.io/projected/b38a7516-2ebf-4d8a-94af-5428d538e584-kube-api-access-9z6fw\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.511321 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.511301    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-credential-socket\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.511395 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.511322    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-data\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.511480 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.511423    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38a7516-2ebf-4d8a-94af-5428d538e584-istiod-ca-cert\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.513107 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.513088    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-envoy\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.513441 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.513424    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-podinfo\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.530445 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.530386    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-token\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.538661 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.538639    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z6fw\" (UniqueName: \"kubernetes.io/projected/b38a7516-2ebf-4d8a-94af-5428d538e584-kube-api-access-9z6fw\") pod \"e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") " pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.612258 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.612230    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6m58\" (UniqueName: \"kubernetes.io/projected/b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a-kube-api-access-n6m58\") pod \"b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a\" (UID: \"b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a\") "
Sep 11 12:02:18.614181 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.614157    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a-kube-api-access-n6m58" (OuterVolumeSpecName: "kube-api-access-n6m58") pod "b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a" (UID: "b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a"). InnerVolumeSpecName "kube-api-access-n6m58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:02:18.633636 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.633607    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:18.714129 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.714097    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6m58\" (UniqueName: \"kubernetes.io/projected/b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a-kube-api-access-n6m58\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:02:18.939449 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:02:18.939421    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38a7516_2ebf_4d8a_94af_5428d538e584.slice/crio-83d17e7403011ad60f985d09e53a609e016bae2e9b51e041cafb862a4997e505 WatchSource:0}: Error finding container 83d17e7403011ad60f985d09e53a609e016bae2e9b51e041cafb862a4997e505: Status 404 returned error can't find the container with id 83d17e7403011ad60f985d09e53a609e016bae2e9b51e041cafb862a4997e505
Sep 11 12:02:18.941781 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:18.941753    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"]
Sep 11 12:02:19.281971 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:19.281935    2569 generic.go:356] "Generic (PLEG): container finished" podID="b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a" containerID="cc705769280f93aad5b99ae6814a499f7fa5799eeb3c2b02e403e989257af9df" exitCode=137
Sep 11 12:02:19.282401 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:19.281983    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7474-249f61"
Sep 11 12:02:19.282401 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:19.282115    2569 scope.go:117] "RemoveContainer" containerID="cc705769280f93aad5b99ae6814a499f7fa5799eeb3c2b02e403e989257af9df"
Sep 11 12:02:19.283967 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:19.283942    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f" event={"ID":"b38a7516-2ebf-4d8a-94af-5428d538e584","Type":"ContainerStarted","Data":"83d17e7403011ad60f985d09e53a609e016bae2e9b51e041cafb862a4997e505"}
Sep 11 12:02:19.292344 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:19.292328    2569 scope.go:117] "RemoveContainer" containerID="cc705769280f93aad5b99ae6814a499f7fa5799eeb3c2b02e403e989257af9df"
Sep 11 12:02:19.292637 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:02:19.292618    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc705769280f93aad5b99ae6814a499f7fa5799eeb3c2b02e403e989257af9df\": container with ID starting with cc705769280f93aad5b99ae6814a499f7fa5799eeb3c2b02e403e989257af9df not found: ID does not exist" containerID="cc705769280f93aad5b99ae6814a499f7fa5799eeb3c2b02e403e989257af9df"
Sep 11 12:02:19.292719 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:19.292645    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc705769280f93aad5b99ae6814a499f7fa5799eeb3c2b02e403e989257af9df"} err="failed to get container status \"cc705769280f93aad5b99ae6814a499f7fa5799eeb3c2b02e403e989257af9df\": rpc error: code = NotFound desc = could not find container \"cc705769280f93aad5b99ae6814a499f7fa5799eeb3c2b02e403e989257af9df\": container with ID starting with cc705769280f93aad5b99ae6814a499f7fa5799eeb3c2b02e403e989257af9df not found: ID does not exist"
Sep 11 12:02:19.296300 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:02:19.296271    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7474-249f61\" is forbidden: User \"system:node:ip-10-0-143-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-220.ec2.internal' and this object" podUID="b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a" pod="opendatahub/test-curl-7474-249f61"
Sep 11 12:02:20.071430 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:20.071389    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a" path="/var/lib/kubelet/pods/b9e6fa6f-87c4-42f8-9ab5-f1284fea7a1a/volumes"
Sep 11 12:02:20.438303 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:20.438279    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7474-b6e1fb"
Sep 11 12:02:20.529666 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:20.529617    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54v7d\" (UniqueName: \"kubernetes.io/projected/d055e321-1bc4-42cf-b336-9e78ea57b484-kube-api-access-54v7d\") pod \"d055e321-1bc4-42cf-b336-9e78ea57b484\" (UID: \"d055e321-1bc4-42cf-b336-9e78ea57b484\") "
Sep 11 12:02:20.531953 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:20.531929    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d055e321-1bc4-42cf-b336-9e78ea57b484-kube-api-access-54v7d" (OuterVolumeSpecName: "kube-api-access-54v7d") pod "d055e321-1bc4-42cf-b336-9e78ea57b484" (UID: "d055e321-1bc4-42cf-b336-9e78ea57b484"). InnerVolumeSpecName "kube-api-access-54v7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:02:20.630423 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:20.630362    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54v7d\" (UniqueName: \"kubernetes.io/projected/d055e321-1bc4-42cf-b336-9e78ea57b484-kube-api-access-54v7d\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:02:21.300535 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:21.300489    2569 generic.go:356] "Generic (PLEG): container finished" podID="d055e321-1bc4-42cf-b336-9e78ea57b484" containerID="23ebeece1dc93bd1786d7643b0c0f3a48b58a23295ba0e5f58f182351d480a1b" exitCode=137
Sep 11 12:02:21.300745 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:21.300550    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7474-b6e1fb"
Sep 11 12:02:21.300745 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:21.300630    2569 scope.go:117] "RemoveContainer" containerID="23ebeece1dc93bd1786d7643b0c0f3a48b58a23295ba0e5f58f182351d480a1b"
Sep 11 12:02:21.315164 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:02:21.315126    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7474-b6e1fb\" is forbidden: User \"system:node:ip-10-0-143-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-220.ec2.internal' and this object" podUID="d055e321-1bc4-42cf-b336-9e78ea57b484" pod="opendatahub/test-curl-7474-b6e1fb"
Sep 11 12:02:21.795176 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:21.795155    2569 scope.go:117] "RemoveContainer" containerID="23ebeece1dc93bd1786d7643b0c0f3a48b58a23295ba0e5f58f182351d480a1b"
Sep 11 12:02:21.795529 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:02:21.795507    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ebeece1dc93bd1786d7643b0c0f3a48b58a23295ba0e5f58f182351d480a1b\": container with ID starting with 23ebeece1dc93bd1786d7643b0c0f3a48b58a23295ba0e5f58f182351d480a1b not found: ID does not exist" containerID="23ebeece1dc93bd1786d7643b0c0f3a48b58a23295ba0e5f58f182351d480a1b"
Sep 11 12:02:21.795572 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:21.795539    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ebeece1dc93bd1786d7643b0c0f3a48b58a23295ba0e5f58f182351d480a1b"} err="failed to get container status \"23ebeece1dc93bd1786d7643b0c0f3a48b58a23295ba0e5f58f182351d480a1b\": rpc error: code = NotFound desc = could not find container \"23ebeece1dc93bd1786d7643b0c0f3a48b58a23295ba0e5f58f182351d480a1b\": container with ID starting with 23ebeece1dc93bd1786d7643b0c0f3a48b58a23295ba0e5f58f182351d480a1b not found: ID does not exist"
Sep 11 12:02:21.857511 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:21.857476    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 12:02:21.857597 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:21.857547    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 12:02:21.857597 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:21.857575    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 12:02:22.069971 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:22.069899    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d055e321-1bc4-42cf-b336-9e78ea57b484" path="/var/lib/kubelet/pods/d055e321-1bc4-42cf-b336-9e78ea57b484/volumes"
Sep 11 12:02:22.308199 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:22.308161    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f" event={"ID":"b38a7516-2ebf-4d8a-94af-5428d538e584","Type":"ContainerStarted","Data":"316cae16c4065f6dece5970b04f0ac370d633567663655089a601c390b07d37c"}
Sep 11 12:02:22.341961 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:22.341877    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f" podStartSLOduration=1.426250327 podStartE2EDuration="4.341860151s" totalImagesPullingTime="2.915609824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:02:22.339335074 +0000 UTC m=+864.761203464" watchObservedRunningTime="2026-09-11 12:02:22.341860151 +0000 UTC m=+864.763728574"
Sep 11 12:02:22.634472 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:22.634398    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:22.639605 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:22.639570    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:23.315076 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:23.315040    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:23.316050 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:23.316028    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:23.541680 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:23.541650    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"]
Sep 11 12:02:23.545466 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:23.545446    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"
Sep 11 12:02:23.549958 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:23.549943    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-cdec04-dockercfg-zhhtl\""
Sep 11 12:02:23.550441 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:23.550428    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-cdec04\""
Sep 11 12:02:23.563010 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:23.562983    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"]
Sep 11 12:02:23.662274 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:23.662196    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1accd26-cf3c-4538-8773-bb9ac5090e3f-plugins-config-volume\") pod \"payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c\" (UID: \"b1accd26-cf3c-4538-8773-bb9ac5090e3f\") " pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"
Sep 11 12:02:23.662440 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:23.662354    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl9mq\" (UniqueName: \"kubernetes.io/projected/b1accd26-cf3c-4538-8773-bb9ac5090e3f-kube-api-access-xl9mq\") pod \"payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c\" (UID: \"b1accd26-cf3c-4538-8773-bb9ac5090e3f\") " pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"
Sep 11 12:02:23.763259 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:23.763222    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1accd26-cf3c-4538-8773-bb9ac5090e3f-plugins-config-volume\") pod \"payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c\" (UID: \"b1accd26-cf3c-4538-8773-bb9ac5090e3f\") " pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"
Sep 11 12:02:23.763445 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:23.763345    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl9mq\" (UniqueName: \"kubernetes.io/projected/b1accd26-cf3c-4538-8773-bb9ac5090e3f-kube-api-access-xl9mq\") pod \"payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c\" (UID: \"b1accd26-cf3c-4538-8773-bb9ac5090e3f\") " pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"
Sep 11 12:02:23.764035 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:23.764012    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1accd26-cf3c-4538-8773-bb9ac5090e3f-plugins-config-volume\") pod \"payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c\" (UID: \"b1accd26-cf3c-4538-8773-bb9ac5090e3f\") " pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"
Sep 11 12:02:23.775821 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:23.775796    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl9mq\" (UniqueName: \"kubernetes.io/projected/b1accd26-cf3c-4538-8773-bb9ac5090e3f-kube-api-access-xl9mq\") pod \"payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c\" (UID: \"b1accd26-cf3c-4538-8773-bb9ac5090e3f\") " pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"
Sep 11 12:02:23.855656 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:23.855627    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"
Sep 11 12:02:24.148851 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:24.148828    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"]
Sep 11 12:02:24.151059 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:02:24.151020    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1accd26_cf3c_4538_8773_bb9ac5090e3f.slice/crio-c7da48023413af0a766b5151256863143dd9fc30df05b1a5aa9ce58f0197ab76 WatchSource:0}: Error finding container c7da48023413af0a766b5151256863143dd9fc30df05b1a5aa9ce58f0197ab76: Status 404 returned error can't find the container with id c7da48023413af0a766b5151256863143dd9fc30df05b1a5aa9ce58f0197ab76
Sep 11 12:02:24.321685 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:24.321649    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c" event={"ID":"b1accd26-cf3c-4538-8773-bb9ac5090e3f","Type":"ContainerStarted","Data":"529b16a49516ff8459f3e854edbf268e908537ae48f9b04a52cdfa9c186f595e"}
Sep 11 12:02:24.322091 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:24.321701    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c" event={"ID":"b1accd26-cf3c-4538-8773-bb9ac5090e3f","Type":"ContainerStarted","Data":"c7da48023413af0a766b5151256863143dd9fc30df05b1a5aa9ce58f0197ab76"}
Sep 11 12:02:24.322091 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:24.321728    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"
Sep 11 12:02:24.433762 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:24.433658    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c" podStartSLOduration=1.4336389010000001 podStartE2EDuration="1.433638901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:02:24.430065751 +0000 UTC m=+866.851934132" watchObservedRunningTime="2026-09-11 12:02:24.433638901 +0000 UTC m=+866.855507284"
Sep 11 12:02:25.292716 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:25.292685    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"]
Sep 11 12:02:25.301368 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:25.297408    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:02:25.301368 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:25.300934    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-49887c13-serving-cert\""
Sep 11 12:02:25.301797 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:25.301771    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 12:02:25.302810 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:25.302784    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rrxjh\""
Sep 11 12:02:25.315024 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:25.314997    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"]
Sep 11 12:02:25.478638 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:25.478605    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b780bf82-2e94-4b61-91de-73295b6a70b4-metrics-certs\") pod \"maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4\" (UID: \"b780bf82-2e94-4b61-91de-73295b6a70b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:02:25.479081 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:25.478650    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fjcg\" (UniqueName: \"kubernetes.io/projected/b780bf82-2e94-4b61-91de-73295b6a70b4-kube-api-access-8fjcg\") pod \"maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4\" (UID: \"b780bf82-2e94-4b61-91de-73295b6a70b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:02:25.479081 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:25.478832    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b780bf82-2e94-4b61-91de-73295b6a70b4-openshift-service-ca\") pod \"maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4\" (UID: \"b780bf82-2e94-4b61-91de-73295b6a70b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:02:25.479081 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:25.478899    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b780bf82-2e94-4b61-91de-73295b6a70b4-maas-api-tls\") pod \"maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4\" (UID: \"b780bf82-2e94-4b61-91de-73295b6a70b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:02:25.580150 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:25.580074    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b780bf82-2e94-4b61-91de-73295b6a70b4-metrics-certs\") pod \"maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4\" (UID: \"b780bf82-2e94-4b61-91de-73295b6a70b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:02:25.580150 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:25.580115    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fjcg\" (UniqueName: \"kubernetes.io/projected/b780bf82-2e94-4b61-91de-73295b6a70b4-kube-api-access-8fjcg\") pod \"maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4\" (UID: \"b780bf82-2e94-4b61-91de-73295b6a70b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:02:25.580324 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:25.580155    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b780bf82-2e94-4b61-91de-73295b6a70b4-openshift-service-ca\") pod \"maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4\" (UID: \"b780bf82-2e94-4b61-91de-73295b6a70b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:02:25.580324 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:25.580174    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b780bf82-2e94-4b61-91de-73295b6a70b4-maas-api-tls\") pod \"maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4\" (UID: \"b780bf82-2e94-4b61-91de-73295b6a70b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:02:25.580324 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:02:25.580254    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-serving-cert: secret "maas-api-e2e-ait-49887c13-serving-cert" not found
Sep 11 12:02:25.580324 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:02:25.580310    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b780bf82-2e94-4b61-91de-73295b6a70b4-maas-api-tls podName:b780bf82-2e94-4b61-91de-73295b6a70b4 nodeName:}" failed. No retries permitted until 2026-09-11 12:02:26.080292973 +0000 UTC m=+868.502161351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b780bf82-2e94-4b61-91de-73295b6a70b4-maas-api-tls") pod "maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4" (UID: "b780bf82-2e94-4b61-91de-73295b6a70b4") : secret "maas-api-e2e-ait-49887c13-serving-cert" not found
Sep 11 12:02:25.580955 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:25.580932    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b780bf82-2e94-4b61-91de-73295b6a70b4-openshift-service-ca\") pod \"maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4\" (UID: \"b780bf82-2e94-4b61-91de-73295b6a70b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:02:25.582489 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:25.582472    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b780bf82-2e94-4b61-91de-73295b6a70b4-metrics-certs\") pod \"maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4\" (UID: \"b780bf82-2e94-4b61-91de-73295b6a70b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:02:25.596522 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:25.596502    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fjcg\" (UniqueName: \"kubernetes.io/projected/b780bf82-2e94-4b61-91de-73295b6a70b4-kube-api-access-8fjcg\") pod \"maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4\" (UID: \"b780bf82-2e94-4b61-91de-73295b6a70b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:02:26.086430 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:26.086395    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b780bf82-2e94-4b61-91de-73295b6a70b4-maas-api-tls\") pod \"maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4\" (UID: \"b780bf82-2e94-4b61-91de-73295b6a70b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:02:26.088872 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:26.088846    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b780bf82-2e94-4b61-91de-73295b6a70b4-maas-api-tls\") pod \"maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4\" (UID: \"b780bf82-2e94-4b61-91de-73295b6a70b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:02:26.211552 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:26.211517    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:02:26.492581 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:26.492555    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"]
Sep 11 12:02:26.493555 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:02:26.493530    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb780bf82_2e94_4b61_91de_73295b6a70b4.slice/crio-782e6aa4fab4f63dd33b38075b07eaca29a478dc5ad883ccc4680bcebb230b91 WatchSource:0}: Error finding container 782e6aa4fab4f63dd33b38075b07eaca29a478dc5ad883ccc4680bcebb230b91: Status 404 returned error can't find the container with id 782e6aa4fab4f63dd33b38075b07eaca29a478dc5ad883ccc4680bcebb230b91
Sep 11 12:02:27.340159 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:27.340123    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4" event={"ID":"b780bf82-2e94-4b61-91de-73295b6a70b4","Type":"ContainerStarted","Data":"782e6aa4fab4f63dd33b38075b07eaca29a478dc5ad883ccc4680bcebb230b91"}
Sep 11 12:02:30.360087 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:30.360052    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4" event={"ID":"b780bf82-2e94-4b61-91de-73295b6a70b4","Type":"ContainerStarted","Data":"0c2632531d4ce6e66fd50ef554da77730d1cc798bac43049588f71a95027860e"}
Sep 11 12:02:30.360437 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:30.360116    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:02:30.393813 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:30.393766    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4" podStartSLOduration=2.390522839 podStartE2EDuration="5.393751728s" totalImagesPullingTime="3.003228889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:02:30.390134699 +0000 UTC m=+872.812003080" watchObservedRunningTime="2026-09-11 12:02:30.393751728 +0000 UTC m=+872.815620107"
Sep 11 12:02:35.329782 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:35.329755    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"
Sep 11 12:02:36.371704 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:36.371627    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:02:48.465934 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:48.465856    2569 generic.go:356] "Generic (PLEG): container finished" podID="b38a7516-2ebf-4d8a-94af-5428d538e584" containerID="316cae16c4065f6dece5970b04f0ac370d633567663655089a601c390b07d37c" exitCode=0
Sep 11 12:02:48.466313 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:48.465937    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f" event={"ID":"b38a7516-2ebf-4d8a-94af-5428d538e584","Type":"ContainerDied","Data":"316cae16c4065f6dece5970b04f0ac370d633567663655089a601c390b07d37c"}
Sep 11 12:02:48.466368 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:48.466353    2569 scope.go:117] "RemoveContainer" containerID="316cae16c4065f6dece5970b04f0ac370d633567663655089a601c390b07d37c"
Sep 11 12:02:48.467435 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:48.467407    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 12:02:48.467488 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:48.467474    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 12:02:48.467536 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:48.467525    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 12:02:49.202330 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.202299    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"]
Sep 11 12:02:49.204831 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.204816    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"
Sep 11 12:02:49.207631 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.207609    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-12d2a9-dockercfg-x9xfk\""
Sep 11 12:02:49.208135 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.208117    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-12d2a9\""
Sep 11 12:02:49.222722 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.222689    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"]
Sep 11 12:02:49.308298 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.308260    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv5zk\" (UniqueName: \"kubernetes.io/projected/dadfdeb4-ac33-44f0-bec2-9ff59086bb21-kube-api-access-nv5zk\") pod \"payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq\" (UID: \"dadfdeb4-ac33-44f0-bec2-9ff59086bb21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"
Sep 11 12:02:49.308469 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.308302    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dadfdeb4-ac33-44f0-bec2-9ff59086bb21-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq\" (UID: \"dadfdeb4-ac33-44f0-bec2-9ff59086bb21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"
Sep 11 12:02:49.372069 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.372033    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49887c13-qxd9f"]
Sep 11 12:02:49.375087 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.375067    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49887c13-qxd9f"
Sep 11 12:02:49.381935 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.381909    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xkpf\""
Sep 11 12:02:49.409055 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.409030    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv5zk\" (UniqueName: \"kubernetes.io/projected/dadfdeb4-ac33-44f0-bec2-9ff59086bb21-kube-api-access-nv5zk\") pod \"payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq\" (UID: \"dadfdeb4-ac33-44f0-bec2-9ff59086bb21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"
Sep 11 12:02:49.409174 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.409071    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dadfdeb4-ac33-44f0-bec2-9ff59086bb21-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq\" (UID: \"dadfdeb4-ac33-44f0-bec2-9ff59086bb21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"
Sep 11 12:02:49.410171 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.410150    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dadfdeb4-ac33-44f0-bec2-9ff59086bb21-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq\" (UID: \"dadfdeb4-ac33-44f0-bec2-9ff59086bb21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"
Sep 11 12:02:49.413617 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.413579    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49887c13-qxd9f"]
Sep 11 12:02:49.431734 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.431711    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv5zk\" (UniqueName: \"kubernetes.io/projected/dadfdeb4-ac33-44f0-bec2-9ff59086bb21-kube-api-access-nv5zk\") pod \"payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq\" (UID: \"dadfdeb4-ac33-44f0-bec2-9ff59086bb21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"
Sep 11 12:02:49.474990 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.474904    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f" event={"ID":"b38a7516-2ebf-4d8a-94af-5428d538e584","Type":"ContainerStarted","Data":"59eabe0f178398dbf1ddbc612829fc28015bec79feafdb849d023792bd6286de"}
Sep 11 12:02:49.509889 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.509859    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7303a6d-63ae-4d3e-a54a-509da95b65e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-49887c13-qxd9f\" (UID: \"f7303a6d-63ae-4d3e-a54a-509da95b65e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49887c13-qxd9f"
Sep 11 12:02:49.515757 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.515732    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"
Sep 11 12:02:49.611470 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.611080    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7303a6d-63ae-4d3e-a54a-509da95b65e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-49887c13-qxd9f\" (UID: \"f7303a6d-63ae-4d3e-a54a-509da95b65e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49887c13-qxd9f"
Sep 11 12:02:49.611836 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.611814    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7303a6d-63ae-4d3e-a54a-509da95b65e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-49887c13-qxd9f\" (UID: \"f7303a6d-63ae-4d3e-a54a-509da95b65e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49887c13-qxd9f"
Sep 11 12:02:49.633901 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.633869    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:49.634693 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.634673    2569 patch_prober.go:28] interesting pod/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 11 12:02:49.634766 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.634716    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f" podUID="b38a7516-2ebf-4d8a-94af-5428d538e584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 11 12:02:49.685359 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.685315    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49887c13-qxd9f"
Sep 11 12:02:49.818004 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.817977    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"]
Sep 11 12:02:49.819719 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:02:49.819692    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddadfdeb4_ac33_44f0_bec2_9ff59086bb21.slice/crio-cf07bab06f0c8409f59c8a69a38e54cfd04379ac87c1a403a43fd359368b9b60 WatchSource:0}: Error finding container cf07bab06f0c8409f59c8a69a38e54cfd04379ac87c1a403a43fd359368b9b60: Status 404 returned error can't find the container with id cf07bab06f0c8409f59c8a69a38e54cfd04379ac87c1a403a43fd359368b9b60
Sep 11 12:02:49.982775 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:49.982747    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49887c13-qxd9f"]
Sep 11 12:02:49.984735 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:02:49.984698    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7303a6d_63ae_4d3e_a54a_509da95b65e2.slice/crio-5272ae5b47a5ad3e822f4eff35aea65819ef29b15846b8c1c19406814b94875b WatchSource:0}: Error finding container 5272ae5b47a5ad3e822f4eff35aea65819ef29b15846b8c1c19406814b94875b: Status 404 returned error can't find the container with id 5272ae5b47a5ad3e822f4eff35aea65819ef29b15846b8c1c19406814b94875b
Sep 11 12:02:50.482259 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:50.482225    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49887c13-qxd9f" event={"ID":"f7303a6d-63ae-4d3e-a54a-509da95b65e2","Type":"ContainerStarted","Data":"5272ae5b47a5ad3e822f4eff35aea65819ef29b15846b8c1c19406814b94875b"}
Sep 11 12:02:50.484048 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:50.484021    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq" event={"ID":"dadfdeb4-ac33-44f0-bec2-9ff59086bb21","Type":"ContainerStarted","Data":"67c9f13371cffe2615ee73f1159397b8dfdf8513ffb2223706b273029396d416"}
Sep 11 12:02:50.484180 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:50.484070    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq" event={"ID":"dadfdeb4-ac33-44f0-bec2-9ff59086bb21","Type":"ContainerStarted","Data":"cf07bab06f0c8409f59c8a69a38e54cfd04379ac87c1a403a43fd359368b9b60"}
Sep 11 12:02:50.484180 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:50.484144    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"
Sep 11 12:02:50.511834 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:50.511786    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq" podStartSLOduration=1.5117727699999999 podStartE2EDuration="1.51177277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:02:50.509828359 +0000 UTC m=+892.931696758" watchObservedRunningTime="2026-09-11 12:02:50.51177277 +0000 UTC m=+892.933641194"
Sep 11 12:02:50.634683 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:50.634649    2569 patch_prober.go:28] interesting pod/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 11 12:02:50.634820 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:50.634699    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f" podUID="b38a7516-2ebf-4d8a-94af-5428d538e584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 11 12:02:51.634270 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:51.634174    2569 patch_prober.go:28] interesting pod/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 11 12:02:51.634270 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:51.634246    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f" podUID="b38a7516-2ebf-4d8a-94af-5428d538e584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 11 12:02:52.634207 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:52.634174    2569 patch_prober.go:28] interesting pod/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 11 12:02:52.634326 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:52.634225    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f" podUID="b38a7516-2ebf-4d8a-94af-5428d538e584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 11 12:02:53.315918 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:53.315884    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:53.505457 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:53.505424    2569 generic.go:356] "Generic (PLEG): container finished" podID="f7303a6d-63ae-4d3e-a54a-509da95b65e2" containerID="3bdd0646be07d164c971e8a7aaaca4c81d503df89665e0cfaee35e1defb8fe96" exitCode=0
Sep 11 12:02:53.505654 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:53.505482    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49887c13-qxd9f" event={"ID":"f7303a6d-63ae-4d3e-a54a-509da95b65e2","Type":"ContainerDied","Data":"3bdd0646be07d164c971e8a7aaaca4c81d503df89665e0cfaee35e1defb8fe96"}
Sep 11 12:02:53.634425 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:53.634338    2569 patch_prober.go:28] interesting pod/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 11 12:02:53.634425 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:53.634391    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f" podUID="b38a7516-2ebf-4d8a-94af-5428d538e584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 11 12:02:54.634091 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:54.634054    2569 patch_prober.go:28] interesting pod/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 11 12:02:54.634266 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:54.634135    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f" podUID="b38a7516-2ebf-4d8a-94af-5428d538e584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 11 12:02:54.675211 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:54.675190    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49887c13-qxd9f"
Sep 11 12:02:54.763044 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:54.762955    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7303a6d-63ae-4d3e-a54a-509da95b65e2-maas-api-service-ca\") pod \"f7303a6d-63ae-4d3e-a54a-509da95b65e2\" (UID: \"f7303a6d-63ae-4d3e-a54a-509da95b65e2\") "
Sep 11 12:02:54.763468 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:54.763437    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7303a6d-63ae-4d3e-a54a-509da95b65e2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7303a6d-63ae-4d3e-a54a-509da95b65e2" (UID: "f7303a6d-63ae-4d3e-a54a-509da95b65e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:02:54.864543 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:54.864503    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7303a6d-63ae-4d3e-a54a-509da95b65e2-maas-api-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:02:55.517419 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:55.517389    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49887c13-qxd9f"
Sep 11 12:02:55.517650 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:55.517425    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49887c13-qxd9f" event={"ID":"f7303a6d-63ae-4d3e-a54a-509da95b65e2","Type":"ContainerDied","Data":"5272ae5b47a5ad3e822f4eff35aea65819ef29b15846b8c1c19406814b94875b"}
Sep 11 12:02:55.517650 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:55.517469    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5272ae5b47a5ad3e822f4eff35aea65819ef29b15846b8c1c19406814b94875b"
Sep 11 12:02:55.638811 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:55.638786    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:55.639646 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:55.639627    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:02:58.100735 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:58.100706    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 12:02:58.101156 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:58.100706    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 12:02:59.380943 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.380906    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"]
Sep 11 12:02:59.381423 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.381404    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7303a6d-63ae-4d3e-a54a-509da95b65e2" containerName="revoke-keys"
Sep 11 12:02:59.381469 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.381425    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7303a6d-63ae-4d3e-a54a-509da95b65e2" containerName="revoke-keys"
Sep 11 12:02:59.381508 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.381502    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7303a6d-63ae-4d3e-a54a-509da95b65e2" containerName="revoke-keys"
Sep 11 12:02:59.383520 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.383505    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.387241 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.387221    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-f67d8993-kserve-self-signed-certs\""
Sep 11 12:02:59.403659 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.403636    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"]
Sep 11 12:02:59.510931 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.510898    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wdw6\" (UniqueName: \"kubernetes.io/projected/7b482ce5-cda8-4696-a3e0-ec196382c0e1-kube-api-access-5wdw6\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.511100 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.510939    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-tmp-dir\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.511100 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.510997    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b482ce5-cda8-4696-a3e0-ec196382c0e1-tls-certs\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.511100 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.511043    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-model-cache\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.511214 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.511122    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-dshm\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.511214 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.511152    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-home\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.612175 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.612144    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-model-cache\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.612393 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.612205    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-dshm\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.612393 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.612223    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-home\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.612393 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.612244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wdw6\" (UniqueName: \"kubernetes.io/projected/7b482ce5-cda8-4696-a3e0-ec196382c0e1-kube-api-access-5wdw6\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.612393 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.612268    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-tmp-dir\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.612393 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.612311    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b482ce5-cda8-4696-a3e0-ec196382c0e1-tls-certs\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.612703 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.612678    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-model-cache\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.612766 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.612691    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-home\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.612823 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.612804    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-tmp-dir\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.614616 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.614577    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-dshm\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.615041 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.615019    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b482ce5-cda8-4696-a3e0-ec196382c0e1-tls-certs\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.622697 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.622675    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wdw6\" (UniqueName: \"kubernetes.io/projected/7b482ce5-cda8-4696-a3e0-ec196382c0e1-kube-api-access-5wdw6\") pod \"e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") " pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.692917 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.692861    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:02:59.969996 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:02:59.969966    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"]
Sep 11 12:02:59.971139 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:02:59.971116    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b482ce5_cda8_4696_a3e0_ec196382c0e1.slice/crio-31a44f9de58465ded1e24cc3ca21a7234f7f83eec4e36a6a77f3b784e8cff1b9 WatchSource:0}: Error finding container 31a44f9de58465ded1e24cc3ca21a7234f7f83eec4e36a6a77f3b784e8cff1b9: Status 404 returned error can't find the container with id 31a44f9de58465ded1e24cc3ca21a7234f7f83eec4e36a6a77f3b784e8cff1b9
Sep 11 12:03:00.547914 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:00.547873    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg" event={"ID":"7b482ce5-cda8-4696-a3e0-ec196382c0e1","Type":"ContainerStarted","Data":"94f9e6a32ec2396549c805429470d5987d5e434f289e83431413bfa7d6046a17"}
Sep 11 12:03:00.547914 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:00.547918    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg" event={"ID":"7b482ce5-cda8-4696-a3e0-ec196382c0e1","Type":"ContainerStarted","Data":"31a44f9de58465ded1e24cc3ca21a7234f7f83eec4e36a6a77f3b784e8cff1b9"}
Sep 11 12:03:00.572028 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:00.571982    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg" podStartSLOduration=1.5719677619999999 podStartE2EDuration="1.571967762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:03:00.569157894 +0000 UTC m=+902.991026285" watchObservedRunningTime="2026-09-11 12:03:00.571967762 +0000 UTC m=+902.993836133"
Sep 11 12:03:01.494203 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:01.494168    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"
Sep 11 12:03:06.216830 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.216789    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"]
Sep 11 12:03:06.217296 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.217010    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4" podUID="b780bf82-2e94-4b61-91de-73295b6a70b4" containerName="maas-api" containerID="cri-o://0c2632531d4ce6e66fd50ef554da77730d1cc798bac43049588f71a95027860e" gracePeriod=30
Sep 11 12:03:06.367012 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.366972    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4" podUID="b780bf82-2e94-4b61-91de-73295b6a70b4" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.57:8443/health\": dial tcp 10.133.0.57:8443: connect: connection refused"
Sep 11 12:03:06.462244 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.462222    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:03:06.585431 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.585400    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b780bf82-2e94-4b61-91de-73295b6a70b4-maas-api-tls\") pod \"b780bf82-2e94-4b61-91de-73295b6a70b4\" (UID: \"b780bf82-2e94-4b61-91de-73295b6a70b4\") "
Sep 11 12:03:06.585577 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.585438    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b780bf82-2e94-4b61-91de-73295b6a70b4-metrics-certs\") pod \"b780bf82-2e94-4b61-91de-73295b6a70b4\" (UID: \"b780bf82-2e94-4b61-91de-73295b6a70b4\") "
Sep 11 12:03:06.585577 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.585510    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fjcg\" (UniqueName: \"kubernetes.io/projected/b780bf82-2e94-4b61-91de-73295b6a70b4-kube-api-access-8fjcg\") pod \"b780bf82-2e94-4b61-91de-73295b6a70b4\" (UID: \"b780bf82-2e94-4b61-91de-73295b6a70b4\") "
Sep 11 12:03:06.585577 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.585550    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b780bf82-2e94-4b61-91de-73295b6a70b4-openshift-service-ca\") pod \"b780bf82-2e94-4b61-91de-73295b6a70b4\" (UID: \"b780bf82-2e94-4b61-91de-73295b6a70b4\") "
Sep 11 12:03:06.586043 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.586008    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b780bf82-2e94-4b61-91de-73295b6a70b4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b780bf82-2e94-4b61-91de-73295b6a70b4" (UID: "b780bf82-2e94-4b61-91de-73295b6a70b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:03:06.587365 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.587334    2569 generic.go:356] "Generic (PLEG): container finished" podID="b780bf82-2e94-4b61-91de-73295b6a70b4" containerID="0c2632531d4ce6e66fd50ef554da77730d1cc798bac43049588f71a95027860e" exitCode=0
Sep 11 12:03:06.587486 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.587379    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"
Sep 11 12:03:06.587486 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.587423    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4" event={"ID":"b780bf82-2e94-4b61-91de-73295b6a70b4","Type":"ContainerDied","Data":"0c2632531d4ce6e66fd50ef554da77730d1cc798bac43049588f71a95027860e"}
Sep 11 12:03:06.587624 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.587485    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4" event={"ID":"b780bf82-2e94-4b61-91de-73295b6a70b4","Type":"ContainerDied","Data":"782e6aa4fab4f63dd33b38075b07eaca29a478dc5ad883ccc4680bcebb230b91"}
Sep 11 12:03:06.587624 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.587518    2569 scope.go:117] "RemoveContainer" containerID="0c2632531d4ce6e66fd50ef554da77730d1cc798bac43049588f71a95027860e"
Sep 11 12:03:06.587941 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.587914    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b780bf82-2e94-4b61-91de-73295b6a70b4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b780bf82-2e94-4b61-91de-73295b6a70b4" (UID: "b780bf82-2e94-4b61-91de-73295b6a70b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:03:06.588055 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.587970    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b780bf82-2e94-4b61-91de-73295b6a70b4-kube-api-access-8fjcg" (OuterVolumeSpecName: "kube-api-access-8fjcg") pod "b780bf82-2e94-4b61-91de-73295b6a70b4" (UID: "b780bf82-2e94-4b61-91de-73295b6a70b4"). InnerVolumeSpecName "kube-api-access-8fjcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:03:06.588055 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.588037    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b780bf82-2e94-4b61-91de-73295b6a70b4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b780bf82-2e94-4b61-91de-73295b6a70b4" (UID: "b780bf82-2e94-4b61-91de-73295b6a70b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:03:06.602745 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.602725    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49887c13-qxd9f"]
Sep 11 12:03:06.607854 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.607833    2569 scope.go:117] "RemoveContainer" containerID="0c2632531d4ce6e66fd50ef554da77730d1cc798bac43049588f71a95027860e"
Sep 11 12:03:06.608143 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:03:06.608122    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2632531d4ce6e66fd50ef554da77730d1cc798bac43049588f71a95027860e\": container with ID starting with 0c2632531d4ce6e66fd50ef554da77730d1cc798bac43049588f71a95027860e not found: ID does not exist" containerID="0c2632531d4ce6e66fd50ef554da77730d1cc798bac43049588f71a95027860e"
Sep 11 12:03:06.608197 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.608153    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2632531d4ce6e66fd50ef554da77730d1cc798bac43049588f71a95027860e"} err="failed to get container status \"0c2632531d4ce6e66fd50ef554da77730d1cc798bac43049588f71a95027860e\": rpc error: code = NotFound desc = could not find container \"0c2632531d4ce6e66fd50ef554da77730d1cc798bac43049588f71a95027860e\": container with ID starting with 0c2632531d4ce6e66fd50ef554da77730d1cc798bac43049588f71a95027860e not found: ID does not exist"
Sep 11 12:03:06.612827 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.612804    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49887c13-qxd9f"]
Sep 11 12:03:06.686437 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.686408    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fjcg\" (UniqueName: \"kubernetes.io/projected/b780bf82-2e94-4b61-91de-73295b6a70b4-kube-api-access-8fjcg\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:06.686437 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.686435    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b780bf82-2e94-4b61-91de-73295b6a70b4-openshift-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:06.686673 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.686447    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b780bf82-2e94-4b61-91de-73295b6a70b4-maas-api-tls\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:06.686673 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.686456    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b780bf82-2e94-4b61-91de-73295b6a70b4-metrics-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:06.749595 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.749553    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"]
Sep 11 12:03:06.749802 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.749778    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f" podUID="b38a7516-2ebf-4d8a-94af-5428d538e584" containerName="istio-proxy" containerID="cri-o://59eabe0f178398dbf1ddbc612829fc28015bec79feafdb849d023792bd6286de" gracePeriod=30
Sep 11 12:03:06.923500 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.923469    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"]
Sep 11 12:03:06.931022 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:06.930998    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49887c13-6967fbb6dd-8bhb4"]
Sep 11 12:03:08.071176 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:08.071141    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b780bf82-2e94-4b61-91de-73295b6a70b4" path="/var/lib/kubelet/pods/b780bf82-2e94-4b61-91de-73295b6a70b4/volumes"
Sep 11 12:03:08.071650 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:08.071568    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7303a6d-63ae-4d3e-a54a-509da95b65e2" path="/var/lib/kubelet/pods/f7303a6d-63ae-4d3e-a54a-509da95b65e2/volumes"
Sep 11 12:03:09.694037 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:09.694004    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:03:09.694411 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:09.694047    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:03:09.706310 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:09.706287    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:03:10.627299 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:10.627263    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:03:10.642350 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:10.642318    2569 patch_prober.go:28] interesting pod/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 11 12:03:10.642472 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:10.642364    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f" podUID="b38a7516-2ebf-4d8a-94af-5428d538e584" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 12:03:10.817298 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:10.817264    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"]
Sep 11 12:03:10.818698 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:10.818672    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b780bf82-2e94-4b61-91de-73295b6a70b4" containerName="maas-api"
Sep 11 12:03:10.818843 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:10.818833    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b780bf82-2e94-4b61-91de-73295b6a70b4" containerName="maas-api"
Sep 11 12:03:10.819135 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:10.819120    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b780bf82-2e94-4b61-91de-73295b6a70b4" containerName="maas-api"
Sep 11 12:03:10.824302 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:10.824279    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:10.827120 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:10.827094    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-150d6af2-serving-cert\""
Sep 11 12:03:10.827541 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:10.827515    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 12:03:10.827770 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:10.827751    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rrxjh\""
Sep 11 12:03:10.833513 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:10.831841    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"]
Sep 11 12:03:10.924089 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:10.924011    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/044359ee-d02a-4776-9cea-dbab9d752eb9-openshift-service-ca\") pod \"maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l\" (UID: \"044359ee-d02a-4776-9cea-dbab9d752eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:10.924089 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:10.924064    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnpnn\" (UniqueName: \"kubernetes.io/projected/044359ee-d02a-4776-9cea-dbab9d752eb9-kube-api-access-wnpnn\") pod \"maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l\" (UID: \"044359ee-d02a-4776-9cea-dbab9d752eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:10.924283 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:10.924205    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/044359ee-d02a-4776-9cea-dbab9d752eb9-metrics-certs\") pod \"maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l\" (UID: \"044359ee-d02a-4776-9cea-dbab9d752eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:10.924283 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:10.924273    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044359ee-d02a-4776-9cea-dbab9d752eb9-maas-api-tls\") pod \"maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l\" (UID: \"044359ee-d02a-4776-9cea-dbab9d752eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:11.025298 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:11.025264    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/044359ee-d02a-4776-9cea-dbab9d752eb9-metrics-certs\") pod \"maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l\" (UID: \"044359ee-d02a-4776-9cea-dbab9d752eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:11.025440 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:11.025321    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044359ee-d02a-4776-9cea-dbab9d752eb9-maas-api-tls\") pod \"maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l\" (UID: \"044359ee-d02a-4776-9cea-dbab9d752eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:11.025440 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:11.025388    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/044359ee-d02a-4776-9cea-dbab9d752eb9-openshift-service-ca\") pod \"maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l\" (UID: \"044359ee-d02a-4776-9cea-dbab9d752eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:11.025440 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:03:11.025417    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-serving-cert: secret "maas-api-e2e-migrate-150d6af2-serving-cert" not found
Sep 11 12:03:11.025561 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:03:11.025480    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044359ee-d02a-4776-9cea-dbab9d752eb9-maas-api-tls podName:044359ee-d02a-4776-9cea-dbab9d752eb9 nodeName:}" failed. No retries permitted until 2026-09-11 12:03:11.525463821 +0000 UTC m=+913.947332179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/044359ee-d02a-4776-9cea-dbab9d752eb9-maas-api-tls") pod "maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l" (UID: "044359ee-d02a-4776-9cea-dbab9d752eb9") : secret "maas-api-e2e-migrate-150d6af2-serving-cert" not found
Sep 11 12:03:11.025561 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:11.025422    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnpnn\" (UniqueName: \"kubernetes.io/projected/044359ee-d02a-4776-9cea-dbab9d752eb9-kube-api-access-wnpnn\") pod \"maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l\" (UID: \"044359ee-d02a-4776-9cea-dbab9d752eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:11.026221 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:11.026194    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/044359ee-d02a-4776-9cea-dbab9d752eb9-openshift-service-ca\") pod \"maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l\" (UID: \"044359ee-d02a-4776-9cea-dbab9d752eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:11.027862 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:11.027840    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/044359ee-d02a-4776-9cea-dbab9d752eb9-metrics-certs\") pod \"maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l\" (UID: \"044359ee-d02a-4776-9cea-dbab9d752eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:11.040252 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:11.040228    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnpnn\" (UniqueName: \"kubernetes.io/projected/044359ee-d02a-4776-9cea-dbab9d752eb9-kube-api-access-wnpnn\") pod \"maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l\" (UID: \"044359ee-d02a-4776-9cea-dbab9d752eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:11.528691 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:11.528656    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044359ee-d02a-4776-9cea-dbab9d752eb9-maas-api-tls\") pod \"maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l\" (UID: \"044359ee-d02a-4776-9cea-dbab9d752eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:11.531037 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:11.531016    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044359ee-d02a-4776-9cea-dbab9d752eb9-maas-api-tls\") pod \"maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l\" (UID: \"044359ee-d02a-4776-9cea-dbab9d752eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:11.738842 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:11.738810    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:11.990769 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:11.990744    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:03:12.025617 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.025566    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"]
Sep 11 12:03:12.026185 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:03:12.026158    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044359ee_d02a_4776_9cea_dbab9d752eb9.slice/crio-88bf7fd3a915116f3432e1d8ad3d2177b9c931fcbbd1a05f7d27cd16fb3ff0cd WatchSource:0}: Error finding container 88bf7fd3a915116f3432e1d8ad3d2177b9c931fcbbd1a05f7d27cd16fb3ff0cd: Status 404 returned error can't find the container with id 88bf7fd3a915116f3432e1d8ad3d2177b9c931fcbbd1a05f7d27cd16fb3ff0cd
Sep 11 12:03:12.032471 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.032447    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-podinfo\") pod \"b38a7516-2ebf-4d8a-94af-5428d538e584\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") "
Sep 11 12:03:12.032610 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.032491    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-workload-socket\") pod \"b38a7516-2ebf-4d8a-94af-5428d538e584\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") "
Sep 11 12:03:12.032610 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.032514    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38a7516-2ebf-4d8a-94af-5428d538e584-istiod-ca-cert\") pod \"b38a7516-2ebf-4d8a-94af-5428d538e584\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") "
Sep 11 12:03:12.032610 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.032570    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9z6fw\" (UniqueName: \"kubernetes.io/projected/b38a7516-2ebf-4d8a-94af-5428d538e584-kube-api-access-9z6fw\") pod \"b38a7516-2ebf-4d8a-94af-5428d538e584\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") "
Sep 11 12:03:12.032771 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.032630    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-data\") pod \"b38a7516-2ebf-4d8a-94af-5428d538e584\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") "
Sep 11 12:03:12.032771 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.032660    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-envoy\") pod \"b38a7516-2ebf-4d8a-94af-5428d538e584\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") "
Sep 11 12:03:12.032771 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.032724    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-token\") pod \"b38a7516-2ebf-4d8a-94af-5428d538e584\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") "
Sep 11 12:03:12.032771 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.032763    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-workload-certs\") pod \"b38a7516-2ebf-4d8a-94af-5428d538e584\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") "
Sep 11 12:03:12.032945 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.032771    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b38a7516-2ebf-4d8a-94af-5428d538e584" (UID: "b38a7516-2ebf-4d8a-94af-5428d538e584"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:03:12.032945 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.032830    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-credential-socket\") pod \"b38a7516-2ebf-4d8a-94af-5428d538e584\" (UID: \"b38a7516-2ebf-4d8a-94af-5428d538e584\") "
Sep 11 12:03:12.033048 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.032935    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-data" (OuterVolumeSpecName: "istio-data") pod "b38a7516-2ebf-4d8a-94af-5428d538e584" (UID: "b38a7516-2ebf-4d8a-94af-5428d538e584"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:03:12.033158 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.033136    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b38a7516-2ebf-4d8a-94af-5428d538e584" (UID: "b38a7516-2ebf-4d8a-94af-5428d538e584"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:03:12.033237 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.033194    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-workload-socket\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:12.033237 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.033215    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-data\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:12.033432 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.033313    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b38a7516-2ebf-4d8a-94af-5428d538e584" (UID: "b38a7516-2ebf-4d8a-94af-5428d538e584"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:03:12.033985 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.033884    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38a7516-2ebf-4d8a-94af-5428d538e584-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b38a7516-2ebf-4d8a-94af-5428d538e584" (UID: "b38a7516-2ebf-4d8a-94af-5428d538e584"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:03:12.035374 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.035200    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-token" (OuterVolumeSpecName: "istio-token") pod "b38a7516-2ebf-4d8a-94af-5428d538e584" (UID: "b38a7516-2ebf-4d8a-94af-5428d538e584"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:03:12.035374 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.035315    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b38a7516-2ebf-4d8a-94af-5428d538e584" (UID: "b38a7516-2ebf-4d8a-94af-5428d538e584"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 12:03:12.035883 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.035857    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b38a7516-2ebf-4d8a-94af-5428d538e584" (UID: "b38a7516-2ebf-4d8a-94af-5428d538e584"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:03:12.036126 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.036111    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38a7516-2ebf-4d8a-94af-5428d538e584-kube-api-access-9z6fw" (OuterVolumeSpecName: "kube-api-access-9z6fw") pod "b38a7516-2ebf-4d8a-94af-5428d538e584" (UID: "b38a7516-2ebf-4d8a-94af-5428d538e584"). InnerVolumeSpecName "kube-api-access-9z6fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:03:12.134077 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.134049    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-envoy\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:12.134077 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.134077    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-token\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:12.134254 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.134091    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-workload-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:12.134254 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.134104    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38a7516-2ebf-4d8a-94af-5428d538e584-credential-socket\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:12.134254 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.134117    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38a7516-2ebf-4d8a-94af-5428d538e584-istio-podinfo\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:12.134254 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.134130    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38a7516-2ebf-4d8a-94af-5428d538e584-istiod-ca-cert\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:12.134254 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.134144    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z6fw\" (UniqueName: \"kubernetes.io/projected/b38a7516-2ebf-4d8a-94af-5428d538e584-kube-api-access-9z6fw\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:12.626196 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.626154    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l" event={"ID":"044359ee-d02a-4776-9cea-dbab9d752eb9","Type":"ContainerStarted","Data":"88bf7fd3a915116f3432e1d8ad3d2177b9c931fcbbd1a05f7d27cd16fb3ff0cd"}
Sep 11 12:03:12.627921 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.627893    2569 generic.go:356] "Generic (PLEG): container finished" podID="b38a7516-2ebf-4d8a-94af-5428d538e584" containerID="59eabe0f178398dbf1ddbc612829fc28015bec79feafdb849d023792bd6286de" exitCode=0
Sep 11 12:03:12.628045 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.627943    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"
Sep 11 12:03:12.628045 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.627971    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f" event={"ID":"b38a7516-2ebf-4d8a-94af-5428d538e584","Type":"ContainerDied","Data":"59eabe0f178398dbf1ddbc612829fc28015bec79feafdb849d023792bd6286de"}
Sep 11 12:03:12.628045 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.628019    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f" event={"ID":"b38a7516-2ebf-4d8a-94af-5428d538e584","Type":"ContainerDied","Data":"83d17e7403011ad60f985d09e53a609e016bae2e9b51e041cafb862a4997e505"}
Sep 11 12:03:12.628045 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.628044    2569 scope.go:117] "RemoveContainer" containerID="59eabe0f178398dbf1ddbc612829fc28015bec79feafdb849d023792bd6286de"
Sep 11 12:03:12.637241 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.637222    2569 scope.go:117] "RemoveContainer" containerID="316cae16c4065f6dece5970b04f0ac370d633567663655089a601c390b07d37c"
Sep 11 12:03:12.646219 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.646191    2569 scope.go:117] "RemoveContainer" containerID="59eabe0f178398dbf1ddbc612829fc28015bec79feafdb849d023792bd6286de"
Sep 11 12:03:12.646498 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:03:12.646479    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59eabe0f178398dbf1ddbc612829fc28015bec79feafdb849d023792bd6286de\": container with ID starting with 59eabe0f178398dbf1ddbc612829fc28015bec79feafdb849d023792bd6286de not found: ID does not exist" containerID="59eabe0f178398dbf1ddbc612829fc28015bec79feafdb849d023792bd6286de"
Sep 11 12:03:12.646564 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.646508    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59eabe0f178398dbf1ddbc612829fc28015bec79feafdb849d023792bd6286de"} err="failed to get container status \"59eabe0f178398dbf1ddbc612829fc28015bec79feafdb849d023792bd6286de\": rpc error: code = NotFound desc = could not find container \"59eabe0f178398dbf1ddbc612829fc28015bec79feafdb849d023792bd6286de\": container with ID starting with 59eabe0f178398dbf1ddbc612829fc28015bec79feafdb849d023792bd6286de not found: ID does not exist"
Sep 11 12:03:12.646564 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.646532    2569 scope.go:117] "RemoveContainer" containerID="316cae16c4065f6dece5970b04f0ac370d633567663655089a601c390b07d37c"
Sep 11 12:03:12.646977 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:03:12.646945    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316cae16c4065f6dece5970b04f0ac370d633567663655089a601c390b07d37c\": container with ID starting with 316cae16c4065f6dece5970b04f0ac370d633567663655089a601c390b07d37c not found: ID does not exist" containerID="316cae16c4065f6dece5970b04f0ac370d633567663655089a601c390b07d37c"
Sep 11 12:03:12.647076 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.646983    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316cae16c4065f6dece5970b04f0ac370d633567663655089a601c390b07d37c"} err="failed to get container status \"316cae16c4065f6dece5970b04f0ac370d633567663655089a601c390b07d37c\": rpc error: code = NotFound desc = could not find container \"316cae16c4065f6dece5970b04f0ac370d633567663655089a601c390b07d37c\": container with ID starting with 316cae16c4065f6dece5970b04f0ac370d633567663655089a601c390b07d37c not found: ID does not exist"
Sep 11 12:03:12.649992 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.649973    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"]
Sep 11 12:03:12.656883 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:12.656865    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-49887c13-openshift-default-785654ccb6-qqg9f"]
Sep 11 12:03:13.639800 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:13.639769    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l" event={"ID":"044359ee-d02a-4776-9cea-dbab9d752eb9","Type":"ContainerStarted","Data":"1d88e082f2de6de74c02b8e71fecafb5d87f17c44622c783ae342406d1e13181"}
Sep 11 12:03:13.640203 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:13.639882    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:13.714534 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:13.714487    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l" podStartSLOduration=2.273634689 podStartE2EDuration="3.714472906s" totalImagesPullingTime="1.440838217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:03:13.713397798 +0000 UTC m=+916.135266177" watchObservedRunningTime="2026-09-11 12:03:13.714472906 +0000 UTC m=+916.136341268"
Sep 11 12:03:14.071028 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:14.070996    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b38a7516-2ebf-4d8a-94af-5428d538e584" path="/var/lib/kubelet/pods/b38a7516-2ebf-4d8a-94af-5428d538e584/volumes"
Sep 11 12:03:19.651803 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:19.651772    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:35.240965 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:35.240936    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7"]
Sep 11 12:03:35.241356 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:35.241343    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b38a7516-2ebf-4d8a-94af-5428d538e584" containerName="istio-proxy"
Sep 11 12:03:35.241404 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:35.241357    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b38a7516-2ebf-4d8a-94af-5428d538e584" containerName="istio-proxy"
Sep 11 12:03:35.241404 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:35.241379    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b38a7516-2ebf-4d8a-94af-5428d538e584" containerName="istio-proxy"
Sep 11 12:03:35.241404 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:35.241385    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b38a7516-2ebf-4d8a-94af-5428d538e584" containerName="istio-proxy"
Sep 11 12:03:35.241496 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:35.241443    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b38a7516-2ebf-4d8a-94af-5428d538e584" containerName="istio-proxy"
Sep 11 12:03:35.249917 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:35.249883    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7"
Sep 11 12:03:35.253268 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:35.253240    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xkpf\""
Sep 11 12:03:35.255220 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:35.255137    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7"]
Sep 11 12:03:35.330673 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:35.330645    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/091304a6-0791-4834-a38e-df8bc90dd44b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7\" (UID: \"091304a6-0791-4834-a38e-df8bc90dd44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7"
Sep 11 12:03:35.431687 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:35.431652    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/091304a6-0791-4834-a38e-df8bc90dd44b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7\" (UID: \"091304a6-0791-4834-a38e-df8bc90dd44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7"
Sep 11 12:03:35.432210 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:35.432189    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/091304a6-0791-4834-a38e-df8bc90dd44b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7\" (UID: \"091304a6-0791-4834-a38e-df8bc90dd44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7"
Sep 11 12:03:35.562858 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:35.562828    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7"
Sep 11 12:03:35.837529 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:35.837497    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7"]
Sep 11 12:03:35.838451 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:03:35.838424    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091304a6_0791_4834_a38e_df8bc90dd44b.slice/crio-d80c8eca3a08c4cd71fdbe039b5112854d620a1066323207be5a642eef8c3af4 WatchSource:0}: Error finding container d80c8eca3a08c4cd71fdbe039b5112854d620a1066323207be5a642eef8c3af4: Status 404 returned error can't find the container with id d80c8eca3a08c4cd71fdbe039b5112854d620a1066323207be5a642eef8c3af4
Sep 11 12:03:36.043397 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.043366    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"]
Sep 11 12:03:36.044110 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.044091    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b38a7516-2ebf-4d8a-94af-5428d538e584" containerName="istio-proxy"
Sep 11 12:03:36.047915 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.047895    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.051648 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.051574    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-cdec04\"/\"default-dockercfg-zrnws\""
Sep 11 12:03:36.051775 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.051753    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-cdec04\"/\"openshift-service-ca.crt\""
Sep 11 12:03:36.051867 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.051769    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-cdec04\"/\"autores-model-cdec04-kserve-self-signed-certs\""
Sep 11 12:03:36.051867 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.051811    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-cdec04\"/\"kube-root-ca.crt\""
Sep 11 12:03:36.057892 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.057864    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"]
Sep 11 12:03:36.137072 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.137044    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c2cq\" (UniqueName: \"kubernetes.io/projected/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-kube-api-access-6c2cq\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.137179 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.137080    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-model-cache\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.137179 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.137106    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-home\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.137179 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.137171    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-dshm\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.137318 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.137215    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-tmp-dir\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.137318 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.137261    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-tls-certs\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.237795 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.237769    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2cq\" (UniqueName: \"kubernetes.io/projected/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-kube-api-access-6c2cq\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.237890 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.237797    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-model-cache\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.237890 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.237815    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-home\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.237890 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.237837    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-dshm\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.237890 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.237866    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-tmp-dir\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.238070 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.237896    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-tls-certs\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.238185 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.238164    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-home\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.238245 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.238205    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-model-cache\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.238245 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.238239    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-tmp-dir\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.240160 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.240137    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-dshm\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.240291 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.240254    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-tls-certs\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.250520 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.250492    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2cq\" (UniqueName: \"kubernetes.io/projected/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-kube-api-access-6c2cq\") pod \"autores-model-cdec04-kserve-654cc5f6f9-t7f8n\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") " pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.367138 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.367069    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:36.676395 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.676371    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"]
Sep 11 12:03:36.678208 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:03:36.678180    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fff0481_aad2_4ee7_b82f_5bd37efc7ffc.slice/crio-21d342f3f9041517844394327af20cf492cb8b13c38362746f4b128eb96b3fcf WatchSource:0}: Error finding container 21d342f3f9041517844394327af20cf492cb8b13c38362746f4b128eb96b3fcf: Status 404 returned error can't find the container with id 21d342f3f9041517844394327af20cf492cb8b13c38362746f4b128eb96b3fcf
Sep 11 12:03:36.773831 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.773799    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n" event={"ID":"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc","Type":"ContainerStarted","Data":"fada7176c68abf21d4cb18937e930c3dc2f925f89041b91859ff061e57c2d9d4"}
Sep 11 12:03:36.773986 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.773843    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n" event={"ID":"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc","Type":"ContainerStarted","Data":"21d342f3f9041517844394327af20cf492cb8b13c38362746f4b128eb96b3fcf"}
Sep 11 12:03:36.775514 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.775490    2569 generic.go:356] "Generic (PLEG): container finished" podID="091304a6-0791-4834-a38e-df8bc90dd44b" containerID="0e41355b0adf68670028725fa198df39551cbfd336e6df46114013cf23598ffb" exitCode=0
Sep 11 12:03:36.775623 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.775562    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7" event={"ID":"091304a6-0791-4834-a38e-df8bc90dd44b","Type":"ContainerDied","Data":"0e41355b0adf68670028725fa198df39551cbfd336e6df46114013cf23598ffb"}
Sep 11 12:03:36.775676 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.775621    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7" event={"ID":"091304a6-0791-4834-a38e-df8bc90dd44b","Type":"ContainerStarted","Data":"d80c8eca3a08c4cd71fdbe039b5112854d620a1066323207be5a642eef8c3af4"}
Sep 11 12:03:36.795655 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:36.795561    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n" podStartSLOduration=0.795543991 podStartE2EDuration="795.543991ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:03:36.795040015 +0000 UTC m=+939.216908425" watchObservedRunningTime="2026-09-11 12:03:36.795543991 +0000 UTC m=+939.217412374"
Sep 11 12:03:37.910361 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:37.910334    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7"
Sep 11 12:03:37.955295 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:37.955271    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/091304a6-0791-4834-a38e-df8bc90dd44b-maas-api-service-ca\") pod \"091304a6-0791-4834-a38e-df8bc90dd44b\" (UID: \"091304a6-0791-4834-a38e-df8bc90dd44b\") "
Sep 11 12:03:37.955618 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:37.955594    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091304a6-0791-4834-a38e-df8bc90dd44b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "091304a6-0791-4834-a38e-df8bc90dd44b" (UID: "091304a6-0791-4834-a38e-df8bc90dd44b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:03:38.056170 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:38.056106    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/091304a6-0791-4834-a38e-df8bc90dd44b-maas-api-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:38.789724 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:38.789696    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7"
Sep 11 12:03:38.789902 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:38.789736    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7" event={"ID":"091304a6-0791-4834-a38e-df8bc90dd44b","Type":"ContainerDied","Data":"d80c8eca3a08c4cd71fdbe039b5112854d620a1066323207be5a642eef8c3af4"}
Sep 11 12:03:38.789902 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:38.789788    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d80c8eca3a08c4cd71fdbe039b5112854d620a1066323207be5a642eef8c3af4"
Sep 11 12:03:42.250232 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.250196    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"]
Sep 11 12:03:42.250635 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.250439    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l" podUID="044359ee-d02a-4776-9cea-dbab9d752eb9" containerName="maas-api" containerID="cri-o://1d88e082f2de6de74c02b8e71fecafb5d87f17c44622c783ae342406d1e13181" gracePeriod=30
Sep 11 12:03:42.503726 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.503670    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:42.561522 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.561492    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7"]
Sep 11 12:03:42.564849 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.564825    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-150d6af2-kfgh7"]
Sep 11 12:03:42.596481 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.596456    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnpnn\" (UniqueName: \"kubernetes.io/projected/044359ee-d02a-4776-9cea-dbab9d752eb9-kube-api-access-wnpnn\") pod \"044359ee-d02a-4776-9cea-dbab9d752eb9\" (UID: \"044359ee-d02a-4776-9cea-dbab9d752eb9\") "
Sep 11 12:03:42.596611 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.596490    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/044359ee-d02a-4776-9cea-dbab9d752eb9-metrics-certs\") pod \"044359ee-d02a-4776-9cea-dbab9d752eb9\" (UID: \"044359ee-d02a-4776-9cea-dbab9d752eb9\") "
Sep 11 12:03:42.596682 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.596612    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/044359ee-d02a-4776-9cea-dbab9d752eb9-openshift-service-ca\") pod \"044359ee-d02a-4776-9cea-dbab9d752eb9\" (UID: \"044359ee-d02a-4776-9cea-dbab9d752eb9\") "
Sep 11 12:03:42.596682 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.596652    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044359ee-d02a-4776-9cea-dbab9d752eb9-maas-api-tls\") pod \"044359ee-d02a-4776-9cea-dbab9d752eb9\" (UID: \"044359ee-d02a-4776-9cea-dbab9d752eb9\") "
Sep 11 12:03:42.596972 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.596946    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044359ee-d02a-4776-9cea-dbab9d752eb9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "044359ee-d02a-4776-9cea-dbab9d752eb9" (UID: "044359ee-d02a-4776-9cea-dbab9d752eb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:03:42.598738 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.598718    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/044359ee-d02a-4776-9cea-dbab9d752eb9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "044359ee-d02a-4776-9cea-dbab9d752eb9" (UID: "044359ee-d02a-4776-9cea-dbab9d752eb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:03:42.598809 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.598765    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/044359ee-d02a-4776-9cea-dbab9d752eb9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "044359ee-d02a-4776-9cea-dbab9d752eb9" (UID: "044359ee-d02a-4776-9cea-dbab9d752eb9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:03:42.598857 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.598842    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044359ee-d02a-4776-9cea-dbab9d752eb9-kube-api-access-wnpnn" (OuterVolumeSpecName: "kube-api-access-wnpnn") pod "044359ee-d02a-4776-9cea-dbab9d752eb9" (UID: "044359ee-d02a-4776-9cea-dbab9d752eb9"). InnerVolumeSpecName "kube-api-access-wnpnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:03:42.697754 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.697726    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/044359ee-d02a-4776-9cea-dbab9d752eb9-openshift-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:42.697754 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.697750    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044359ee-d02a-4776-9cea-dbab9d752eb9-maas-api-tls\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:42.697754 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.697759    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnpnn\" (UniqueName: \"kubernetes.io/projected/044359ee-d02a-4776-9cea-dbab9d752eb9-kube-api-access-wnpnn\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:42.698039 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.697768    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/044359ee-d02a-4776-9cea-dbab9d752eb9-metrics-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:03:42.813528 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.813454    2569 generic.go:356] "Generic (PLEG): container finished" podID="044359ee-d02a-4776-9cea-dbab9d752eb9" containerID="1d88e082f2de6de74c02b8e71fecafb5d87f17c44622c783ae342406d1e13181" exitCode=0
Sep 11 12:03:42.813528 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.813503    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"
Sep 11 12:03:42.813736 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.813538    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l" event={"ID":"044359ee-d02a-4776-9cea-dbab9d752eb9","Type":"ContainerDied","Data":"1d88e082f2de6de74c02b8e71fecafb5d87f17c44622c783ae342406d1e13181"}
Sep 11 12:03:42.813736 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.813618    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l" event={"ID":"044359ee-d02a-4776-9cea-dbab9d752eb9","Type":"ContainerDied","Data":"88bf7fd3a915116f3432e1d8ad3d2177b9c931fcbbd1a05f7d27cd16fb3ff0cd"}
Sep 11 12:03:42.813736 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.813649    2569 scope.go:117] "RemoveContainer" containerID="1d88e082f2de6de74c02b8e71fecafb5d87f17c44622c783ae342406d1e13181"
Sep 11 12:03:42.823407 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.823392    2569 scope.go:117] "RemoveContainer" containerID="1d88e082f2de6de74c02b8e71fecafb5d87f17c44622c783ae342406d1e13181"
Sep 11 12:03:42.823707 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:03:42.823683    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d88e082f2de6de74c02b8e71fecafb5d87f17c44622c783ae342406d1e13181\": container with ID starting with 1d88e082f2de6de74c02b8e71fecafb5d87f17c44622c783ae342406d1e13181 not found: ID does not exist" containerID="1d88e082f2de6de74c02b8e71fecafb5d87f17c44622c783ae342406d1e13181"
Sep 11 12:03:42.823770 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.823714    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d88e082f2de6de74c02b8e71fecafb5d87f17c44622c783ae342406d1e13181"} err="failed to get container status \"1d88e082f2de6de74c02b8e71fecafb5d87f17c44622c783ae342406d1e13181\": rpc error: code = NotFound desc = could not find container \"1d88e082f2de6de74c02b8e71fecafb5d87f17c44622c783ae342406d1e13181\": container with ID starting with 1d88e082f2de6de74c02b8e71fecafb5d87f17c44622c783ae342406d1e13181 not found: ID does not exist"
Sep 11 12:03:42.869190 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.869167    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"]
Sep 11 12:03:42.871473 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:42.871454    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-150d6af2-6f8cfc4d4f-bfv4l"]
Sep 11 12:03:44.070393 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:44.070362    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="044359ee-d02a-4776-9cea-dbab9d752eb9" path="/var/lib/kubelet/pods/044359ee-d02a-4776-9cea-dbab9d752eb9/volumes"
Sep 11 12:03:44.070777 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:44.070711    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="091304a6-0791-4834-a38e-df8bc90dd44b" path="/var/lib/kubelet/pods/091304a6-0791-4834-a38e-df8bc90dd44b/volumes"
Sep 11 12:03:46.367683 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:46.367648    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:46.368070 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:46.367693    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:46.380055 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:46.380024    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:46.848699 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:46.848668    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:03:47.365203 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.365171    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"]
Sep 11 12:03:47.365796 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.365781    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="091304a6-0791-4834-a38e-df8bc90dd44b" containerName="revoke-keys"
Sep 11 12:03:47.365848 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.365799    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="091304a6-0791-4834-a38e-df8bc90dd44b" containerName="revoke-keys"
Sep 11 12:03:47.365848 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.365826    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="044359ee-d02a-4776-9cea-dbab9d752eb9" containerName="maas-api"
Sep 11 12:03:47.365848 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.365834    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="044359ee-d02a-4776-9cea-dbab9d752eb9" containerName="maas-api"
Sep 11 12:03:47.365947 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.365936    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="091304a6-0791-4834-a38e-df8bc90dd44b" containerName="revoke-keys"
Sep 11 12:03:47.365987 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.365953    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="044359ee-d02a-4776-9cea-dbab9d752eb9" containerName="maas-api"
Sep 11 12:03:47.369089 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.369071    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:03:47.373020 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.373002    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-087015a2-serving-cert\""
Sep 11 12:03:47.373103 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.373024    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 12:03:47.373850 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.373834    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rrxjh\""
Sep 11 12:03:47.399323 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.399298    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"]
Sep 11 12:03:47.442163 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.442134    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v5zv\" (UniqueName: \"kubernetes.io/projected/733759d5-e428-4ef5-9d03-d8e09b8acd0f-kube-api-access-5v5zv\") pod \"maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9\" (UID: \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:03:47.442286 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.442174    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/733759d5-e428-4ef5-9d03-d8e09b8acd0f-maas-api-tls\") pod \"maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9\" (UID: \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:03:47.442286 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.442204    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733759d5-e428-4ef5-9d03-d8e09b8acd0f-metrics-certs\") pod \"maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9\" (UID: \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:03:47.442286 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.442259    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/733759d5-e428-4ef5-9d03-d8e09b8acd0f-openshift-service-ca\") pod \"maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9\" (UID: \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:03:47.543421 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.543394    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5zv\" (UniqueName: \"kubernetes.io/projected/733759d5-e428-4ef5-9d03-d8e09b8acd0f-kube-api-access-5v5zv\") pod \"maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9\" (UID: \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:03:47.543547 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.543429    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/733759d5-e428-4ef5-9d03-d8e09b8acd0f-maas-api-tls\") pod \"maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9\" (UID: \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:03:47.543547 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.543459    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733759d5-e428-4ef5-9d03-d8e09b8acd0f-metrics-certs\") pod \"maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9\" (UID: \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:03:47.543547 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.543491    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/733759d5-e428-4ef5-9d03-d8e09b8acd0f-openshift-service-ca\") pod \"maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9\" (UID: \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:03:47.543735 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:03:47.543551    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-serving-cert: secret "maas-api-e2e-ait-087015a2-serving-cert" not found
Sep 11 12:03:47.543735 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:03:47.543628    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/733759d5-e428-4ef5-9d03-d8e09b8acd0f-maas-api-tls podName:733759d5-e428-4ef5-9d03-d8e09b8acd0f nodeName:}" failed. No retries permitted until 2026-09-11 12:03:48.043608593 +0000 UTC m=+950.465476972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/733759d5-e428-4ef5-9d03-d8e09b8acd0f-maas-api-tls") pod "maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9" (UID: "733759d5-e428-4ef5-9d03-d8e09b8acd0f") : secret "maas-api-e2e-ait-087015a2-serving-cert" not found
Sep 11 12:03:47.544138 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.544120    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/733759d5-e428-4ef5-9d03-d8e09b8acd0f-openshift-service-ca\") pod \"maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9\" (UID: \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:03:47.545982 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.545961    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733759d5-e428-4ef5-9d03-d8e09b8acd0f-metrics-certs\") pod \"maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9\" (UID: \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:03:47.563650 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:47.563628    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v5zv\" (UniqueName: \"kubernetes.io/projected/733759d5-e428-4ef5-9d03-d8e09b8acd0f-kube-api-access-5v5zv\") pod \"maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9\" (UID: \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:03:48.047377 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:48.047344    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/733759d5-e428-4ef5-9d03-d8e09b8acd0f-maas-api-tls\") pod \"maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9\" (UID: \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:03:48.049740 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:48.049718    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/733759d5-e428-4ef5-9d03-d8e09b8acd0f-maas-api-tls\") pod \"maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9\" (UID: \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:03:48.279578 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:48.279542    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:03:48.564716 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:48.564691    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"]
Sep 11 12:03:48.566645 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:03:48.566619    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733759d5_e428_4ef5_9d03_d8e09b8acd0f.slice/crio-11eff9853c4635e797327aa856bfb9bed70735fd6aa03818819d2df66c1c52b7 WatchSource:0}: Error finding container 11eff9853c4635e797327aa856bfb9bed70735fd6aa03818819d2df66c1c52b7: Status 404 returned error can't find the container with id 11eff9853c4635e797327aa856bfb9bed70735fd6aa03818819d2df66c1c52b7
Sep 11 12:03:48.850880 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:48.850843    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9" event={"ID":"733759d5-e428-4ef5-9d03-d8e09b8acd0f","Type":"ContainerStarted","Data":"11eff9853c4635e797327aa856bfb9bed70735fd6aa03818819d2df66c1c52b7"}
Sep 11 12:03:49.861297 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:49.861261    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9" event={"ID":"733759d5-e428-4ef5-9d03-d8e09b8acd0f","Type":"ContainerStarted","Data":"94159c1c34fe6911eb060c91426ab9c7563d6fb14723d94d30ef31d21cb1cabf"}
Sep 11 12:03:49.861722 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:49.861416    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:03:49.886732 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:49.886687    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9" podStartSLOduration=2.464913061 podStartE2EDuration="2.886672714s" totalImagesPullingTime="421.759653ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:03:49.883560328 +0000 UTC m=+952.305428718" watchObservedRunningTime="2026-09-11 12:03:49.886672714 +0000 UTC m=+952.308541097"
Sep 11 12:03:55.874703 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:03:55.874673    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:04:04.147077 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.147044    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"]
Sep 11 12:04:04.150366 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.150348    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.152887 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.152865    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-f67d8993-kserve-self-signed-certs\""
Sep 11 12:04:04.171059 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.171038    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"]
Sep 11 12:04:04.292013 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.291986    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9c52e16-56a8-424f-bf94-5eebff6c1963-tls-certs\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.292136 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.292026    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-tmp-dir\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.292136 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.292058    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mr28\" (UniqueName: \"kubernetes.io/projected/a9c52e16-56a8-424f-bf94-5eebff6c1963-kube-api-access-4mr28\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.292211 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.292135    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-dshm\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.292211 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.292158    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-model-cache\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.292211 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.292186    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-home\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.393293 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.393255    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-dshm\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.393293 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.393298    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-model-cache\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.393515 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.393327    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-home\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.393515 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.393370    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9c52e16-56a8-424f-bf94-5eebff6c1963-tls-certs\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.393515 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.393410    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-tmp-dir\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.393515 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.393440    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mr28\" (UniqueName: \"kubernetes.io/projected/a9c52e16-56a8-424f-bf94-5eebff6c1963-kube-api-access-4mr28\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.393812 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.393786    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-home\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.393874 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.393813    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-model-cache\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.393874 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.393851    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-tmp-dir\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.395670 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.395651    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-dshm\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.395907 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.395888    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9c52e16-56a8-424f-bf94-5eebff6c1963-tls-certs\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.449204 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.449144    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mr28\" (UniqueName: \"kubernetes.io/projected/a9c52e16-56a8-424f-bf94-5eebff6c1963-kube-api-access-4mr28\") pod \"e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") " pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.460047 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.460022    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:04.738728 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.738703    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"]
Sep 11 12:04:04.739371 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:04:04.739349    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9c52e16_56a8_424f_bf94_5eebff6c1963.slice/crio-03a3e8407578ff5c424dc3f0ac96a164237a0f890bffd2a22c62b2198884f3cf WatchSource:0}: Error finding container 03a3e8407578ff5c424dc3f0ac96a164237a0f890bffd2a22c62b2198884f3cf: Status 404 returned error can't find the container with id 03a3e8407578ff5c424dc3f0ac96a164237a0f890bffd2a22c62b2198884f3cf
Sep 11 12:04:04.952637 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.952602    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd" event={"ID":"a9c52e16-56a8-424f-bf94-5eebff6c1963","Type":"ContainerStarted","Data":"12b0b10b683c93b2a07d7e8c243f26c4fdfabe9deec50a5b597e30cb2a785374"}
Sep 11 12:04:04.952637 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.952643    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd" event={"ID":"a9c52e16-56a8-424f-bf94-5eebff6c1963","Type":"ContainerStarted","Data":"03a3e8407578ff5c424dc3f0ac96a164237a0f890bffd2a22c62b2198884f3cf"}
Sep 11 12:04:04.974550 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:04.974500    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd" podStartSLOduration=0.974484182 podStartE2EDuration="974.484182ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:04:04.972150321 +0000 UTC m=+967.394018701" watchObservedRunningTime="2026-09-11 12:04:04.974484182 +0000 UTC m=+967.396352561"
Sep 11 12:04:09.237232 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:09.237198    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-cdec04-rcrw4"]
Sep 11 12:04:09.240165 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:09.240147    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-cdec04-rcrw4"
Sep 11 12:04:09.243870 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:09.243850    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xkpf\""
Sep 11 12:04:09.254306 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:09.254279    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-cdec04-rcrw4"]
Sep 11 12:04:09.332092 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:09.332061    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e40b1c-872e-4148-a904-2c32c99613f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-cdec04-rcrw4\" (UID: \"d4e40b1c-872e-4148-a904-2c32c99613f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-cdec04-rcrw4"
Sep 11 12:04:09.433038 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:09.432995    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e40b1c-872e-4148-a904-2c32c99613f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-cdec04-rcrw4\" (UID: \"d4e40b1c-872e-4148-a904-2c32c99613f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-cdec04-rcrw4"
Sep 11 12:04:09.433657 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:09.433636    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e40b1c-872e-4148-a904-2c32c99613f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-cdec04-rcrw4\" (UID: \"d4e40b1c-872e-4148-a904-2c32c99613f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-cdec04-rcrw4"
Sep 11 12:04:09.551630 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:09.551606    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-cdec04-rcrw4"
Sep 11 12:04:09.824042 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:09.824007    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-cdec04-rcrw4"]
Sep 11 12:04:09.825668 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:04:09.825638    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e40b1c_872e_4148_a904_2c32c99613f2.slice/crio-6da1eb9626d4389b61d6ee532a2bc2579b102b95598ce227738caf172e7f93f8 WatchSource:0}: Error finding container 6da1eb9626d4389b61d6ee532a2bc2579b102b95598ce227738caf172e7f93f8: Status 404 returned error can't find the container with id 6da1eb9626d4389b61d6ee532a2bc2579b102b95598ce227738caf172e7f93f8
Sep 11 12:04:09.983610 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:09.983560    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-cdec04-rcrw4" event={"ID":"d4e40b1c-872e-4148-a904-2c32c99613f2","Type":"ContainerStarted","Data":"6da1eb9626d4389b61d6ee532a2bc2579b102b95598ce227738caf172e7f93f8"}
Sep 11 12:04:10.990676 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:10.990594    2569 generic.go:356] "Generic (PLEG): container finished" podID="d4e40b1c-872e-4148-a904-2c32c99613f2" containerID="ad6bde0321212c1f42fa2f187805cb45730219b90f1e99f9f176e4244cceea20" exitCode=0
Sep 11 12:04:10.991079 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:10.990688    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-cdec04-rcrw4" event={"ID":"d4e40b1c-872e-4148-a904-2c32c99613f2","Type":"ContainerDied","Data":"ad6bde0321212c1f42fa2f187805cb45730219b90f1e99f9f176e4244cceea20"}
Sep 11 12:04:11.828854 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:11.828818    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-087015a2-dpmvs"]
Sep 11 12:04:11.831370 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:11.831354    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-087015a2-dpmvs"
Sep 11 12:04:11.850241 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:11.850194    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9a7206-e530-4165-8772-1da6e10c026d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-087015a2-dpmvs\" (UID: \"9c9a7206-e530-4165-8772-1da6e10c026d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-087015a2-dpmvs"
Sep 11 12:04:11.854333 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:11.854309    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-087015a2-dpmvs"]
Sep 11 12:04:11.951383 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:11.951353    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9a7206-e530-4165-8772-1da6e10c026d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-087015a2-dpmvs\" (UID: \"9c9a7206-e530-4165-8772-1da6e10c026d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-087015a2-dpmvs"
Sep 11 12:04:11.951921 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:11.951900    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9a7206-e530-4165-8772-1da6e10c026d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-087015a2-dpmvs\" (UID: \"9c9a7206-e530-4165-8772-1da6e10c026d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-087015a2-dpmvs"
Sep 11 12:04:12.124360 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:12.124337    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-cdec04-rcrw4"
Sep 11 12:04:12.141198 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:12.141172    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-087015a2-dpmvs"
Sep 11 12:04:12.153700 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:12.153673    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e40b1c-872e-4148-a904-2c32c99613f2-maas-api-service-ca\") pod \"d4e40b1c-872e-4148-a904-2c32c99613f2\" (UID: \"d4e40b1c-872e-4148-a904-2c32c99613f2\") "
Sep 11 12:04:12.153989 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:12.153970    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e40b1c-872e-4148-a904-2c32c99613f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4e40b1c-872e-4148-a904-2c32c99613f2" (UID: "d4e40b1c-872e-4148-a904-2c32c99613f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:04:12.154069 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:12.154052    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e40b1c-872e-4148-a904-2c32c99613f2-maas-api-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:12.421562 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:12.421537    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-087015a2-dpmvs"]
Sep 11 12:04:12.423214 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:04:12.423187    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9a7206_e530_4165_8772_1da6e10c026d.slice/crio-ae71b180e95c3e42c3e11f9d102ad1b88496e2f4264cc2e74cfbb03a96f9adfc WatchSource:0}: Error finding container ae71b180e95c3e42c3e11f9d102ad1b88496e2f4264cc2e74cfbb03a96f9adfc: Status 404 returned error can't find the container with id ae71b180e95c3e42c3e11f9d102ad1b88496e2f4264cc2e74cfbb03a96f9adfc
Sep 11 12:04:13.003240 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:13.003164    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-cdec04-rcrw4"
Sep 11 12:04:13.003240 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:13.003197    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-cdec04-rcrw4" event={"ID":"d4e40b1c-872e-4148-a904-2c32c99613f2","Type":"ContainerDied","Data":"6da1eb9626d4389b61d6ee532a2bc2579b102b95598ce227738caf172e7f93f8"}
Sep 11 12:04:13.003421 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:13.003244    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6da1eb9626d4389b61d6ee532a2bc2579b102b95598ce227738caf172e7f93f8"
Sep 11 12:04:13.004735 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:13.004714    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-087015a2-dpmvs" event={"ID":"9c9a7206-e530-4165-8772-1da6e10c026d","Type":"ContainerStarted","Data":"b6b1749420337a75e712d30e37528221eb97e21847e666d9c9e3ce57d7e14a9c"}
Sep 11 12:04:13.004846 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:13.004748    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-087015a2-dpmvs" event={"ID":"9c9a7206-e530-4165-8772-1da6e10c026d","Type":"ContainerStarted","Data":"ae71b180e95c3e42c3e11f9d102ad1b88496e2f4264cc2e74cfbb03a96f9adfc"}
Sep 11 12:04:13.046818 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:13.046776    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-087015a2-dpmvs" podStartSLOduration=2.04676307 podStartE2EDuration="2.04676307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:04:13.043422608 +0000 UTC m=+975.465290987" watchObservedRunningTime="2026-09-11 12:04:13.04676307 +0000 UTC m=+975.468631450"
Sep 11 12:04:14.018529 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:14.018494    2569 generic.go:356] "Generic (PLEG): container finished" podID="9c9a7206-e530-4165-8772-1da6e10c026d" containerID="b6b1749420337a75e712d30e37528221eb97e21847e666d9c9e3ce57d7e14a9c" exitCode=0
Sep 11 12:04:14.018980 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:14.018580    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-087015a2-dpmvs" event={"ID":"9c9a7206-e530-4165-8772-1da6e10c026d","Type":"ContainerDied","Data":"b6b1749420337a75e712d30e37528221eb97e21847e666d9c9e3ce57d7e14a9c"}
Sep 11 12:04:14.460821 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:14.460732    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:14.460821 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:14.460778    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:14.473416 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:14.473391    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:15.039740 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:15.039708    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:15.167731 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:15.167708    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-087015a2-dpmvs"
Sep 11 12:04:15.180228 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:15.180208    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9a7206-e530-4165-8772-1da6e10c026d-maas-api-service-ca\") pod \"9c9a7206-e530-4165-8772-1da6e10c026d\" (UID: \"9c9a7206-e530-4165-8772-1da6e10c026d\") "
Sep 11 12:04:15.180515 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:15.180495    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9a7206-e530-4165-8772-1da6e10c026d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c9a7206-e530-4165-8772-1da6e10c026d" (UID: "9c9a7206-e530-4165-8772-1da6e10c026d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:04:15.281213 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:15.281147    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9a7206-e530-4165-8772-1da6e10c026d-maas-api-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:16.031134 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:16.031104    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-087015a2-dpmvs"
Sep 11 12:04:16.031335 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:16.031134    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-087015a2-dpmvs" event={"ID":"9c9a7206-e530-4165-8772-1da6e10c026d","Type":"ContainerDied","Data":"ae71b180e95c3e42c3e11f9d102ad1b88496e2f4264cc2e74cfbb03a96f9adfc"}
Sep 11 12:04:16.031335 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:16.031181    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae71b180e95c3e42c3e11f9d102ad1b88496e2f4264cc2e74cfbb03a96f9adfc"
Sep 11 12:04:18.505931 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:18.505899    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"]
Sep 11 12:04:18.506322 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:18.506204    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9" podUID="733759d5-e428-4ef5-9d03-d8e09b8acd0f" containerName="maas-api" containerID="cri-o://94159c1c34fe6911eb060c91426ab9c7563d6fb14723d94d30ef31d21cb1cabf" gracePeriod=30
Sep 11 12:04:18.763418 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:18.763353    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:04:18.813628 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:18.813578    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/733759d5-e428-4ef5-9d03-d8e09b8acd0f-openshift-service-ca\") pod \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\" (UID: \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\") "
Sep 11 12:04:18.813780 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:18.813650    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/733759d5-e428-4ef5-9d03-d8e09b8acd0f-maas-api-tls\") pod \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\" (UID: \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\") "
Sep 11 12:04:18.813780 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:18.813720    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733759d5-e428-4ef5-9d03-d8e09b8acd0f-metrics-certs\") pod \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\" (UID: \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\") "
Sep 11 12:04:18.813878 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:18.813800    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v5zv\" (UniqueName: \"kubernetes.io/projected/733759d5-e428-4ef5-9d03-d8e09b8acd0f-kube-api-access-5v5zv\") pod \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\" (UID: \"733759d5-e428-4ef5-9d03-d8e09b8acd0f\") "
Sep 11 12:04:18.813943 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:18.813896    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733759d5-e428-4ef5-9d03-d8e09b8acd0f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "733759d5-e428-4ef5-9d03-d8e09b8acd0f" (UID: "733759d5-e428-4ef5-9d03-d8e09b8acd0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:04:18.814130 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:18.814108    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/733759d5-e428-4ef5-9d03-d8e09b8acd0f-openshift-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:18.815706 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:18.815676    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/733759d5-e428-4ef5-9d03-d8e09b8acd0f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "733759d5-e428-4ef5-9d03-d8e09b8acd0f" (UID: "733759d5-e428-4ef5-9d03-d8e09b8acd0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:04:18.815803 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:18.815750    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733759d5-e428-4ef5-9d03-d8e09b8acd0f-kube-api-access-5v5zv" (OuterVolumeSpecName: "kube-api-access-5v5zv") pod "733759d5-e428-4ef5-9d03-d8e09b8acd0f" (UID: "733759d5-e428-4ef5-9d03-d8e09b8acd0f"). InnerVolumeSpecName "kube-api-access-5v5zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:04:18.816046 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:18.816028    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/733759d5-e428-4ef5-9d03-d8e09b8acd0f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "733759d5-e428-4ef5-9d03-d8e09b8acd0f" (UID: "733759d5-e428-4ef5-9d03-d8e09b8acd0f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:04:18.915076 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:18.915039    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733759d5-e428-4ef5-9d03-d8e09b8acd0f-metrics-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:18.915076 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:18.915070    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v5zv\" (UniqueName: \"kubernetes.io/projected/733759d5-e428-4ef5-9d03-d8e09b8acd0f-kube-api-access-5v5zv\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:18.915294 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:18.915084    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/733759d5-e428-4ef5-9d03-d8e09b8acd0f-maas-api-tls\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:19.050528 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:19.050453    2569 generic.go:356] "Generic (PLEG): container finished" podID="733759d5-e428-4ef5-9d03-d8e09b8acd0f" containerID="94159c1c34fe6911eb060c91426ab9c7563d6fb14723d94d30ef31d21cb1cabf" exitCode=0
Sep 11 12:04:19.050528 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:19.050495    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"
Sep 11 12:04:19.050777 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:19.050531    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9" event={"ID":"733759d5-e428-4ef5-9d03-d8e09b8acd0f","Type":"ContainerDied","Data":"94159c1c34fe6911eb060c91426ab9c7563d6fb14723d94d30ef31d21cb1cabf"}
Sep 11 12:04:19.050777 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:19.050581    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9" event={"ID":"733759d5-e428-4ef5-9d03-d8e09b8acd0f","Type":"ContainerDied","Data":"11eff9853c4635e797327aa856bfb9bed70735fd6aa03818819d2df66c1c52b7"}
Sep 11 12:04:19.050777 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:19.050621    2569 scope.go:117] "RemoveContainer" containerID="94159c1c34fe6911eb060c91426ab9c7563d6fb14723d94d30ef31d21cb1cabf"
Sep 11 12:04:19.061246 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:19.061231    2569 scope.go:117] "RemoveContainer" containerID="94159c1c34fe6911eb060c91426ab9c7563d6fb14723d94d30ef31d21cb1cabf"
Sep 11 12:04:19.061508 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:04:19.061490    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94159c1c34fe6911eb060c91426ab9c7563d6fb14723d94d30ef31d21cb1cabf\": container with ID starting with 94159c1c34fe6911eb060c91426ab9c7563d6fb14723d94d30ef31d21cb1cabf not found: ID does not exist" containerID="94159c1c34fe6911eb060c91426ab9c7563d6fb14723d94d30ef31d21cb1cabf"
Sep 11 12:04:19.061554 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:19.061515    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94159c1c34fe6911eb060c91426ab9c7563d6fb14723d94d30ef31d21cb1cabf"} err="failed to get container status \"94159c1c34fe6911eb060c91426ab9c7563d6fb14723d94d30ef31d21cb1cabf\": rpc error: code = NotFound desc = could not find container \"94159c1c34fe6911eb060c91426ab9c7563d6fb14723d94d30ef31d21cb1cabf\": container with ID starting with 94159c1c34fe6911eb060c91426ab9c7563d6fb14723d94d30ef31d21cb1cabf not found: ID does not exist"
Sep 11 12:04:19.126354 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:19.126324    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"]
Sep 11 12:04:19.130880 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:19.130856    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-087015a2-7f97d8cb5c-jcxb9"]
Sep 11 12:04:19.212062 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:19.212033    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-087015a2-dpmvs"]
Sep 11 12:04:19.214353 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:19.214334    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-087015a2-dpmvs"]
Sep 11 12:04:20.070345 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.070314    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="733759d5-e428-4ef5-9d03-d8e09b8acd0f" path="/var/lib/kubelet/pods/733759d5-e428-4ef5-9d03-d8e09b8acd0f/volumes"
Sep 11 12:04:20.070791 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.070775    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c9a7206-e530-4165-8772-1da6e10c026d" path="/var/lib/kubelet/pods/9c9a7206-e530-4165-8772-1da6e10c026d/volumes"
Sep 11 12:04:20.461558 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.461482    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"]
Sep 11 12:04:20.461893 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.461880    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e40b1c-872e-4148-a904-2c32c99613f2" containerName="revoke-keys"
Sep 11 12:04:20.461946 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.461894    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e40b1c-872e-4148-a904-2c32c99613f2" containerName="revoke-keys"
Sep 11 12:04:20.461946 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.461910    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9a7206-e530-4165-8772-1da6e10c026d" containerName="revoke-keys"
Sep 11 12:04:20.461946 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.461916    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9a7206-e530-4165-8772-1da6e10c026d" containerName="revoke-keys"
Sep 11 12:04:20.461946 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.461933    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="733759d5-e428-4ef5-9d03-d8e09b8acd0f" containerName="maas-api"
Sep 11 12:04:20.461946 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.461938    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="733759d5-e428-4ef5-9d03-d8e09b8acd0f" containerName="maas-api"
Sep 11 12:04:20.462096 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.461999    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="733759d5-e428-4ef5-9d03-d8e09b8acd0f" containerName="maas-api"
Sep 11 12:04:20.462096 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.462012    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e40b1c-872e-4148-a904-2c32c99613f2" containerName="revoke-keys"
Sep 11 12:04:20.462096 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.462018    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c9a7206-e530-4165-8772-1da6e10c026d" containerName="revoke-keys"
Sep 11 12:04:20.464952 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.464931    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.472903 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.472880    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cbc380\"/\"kube-root-ca.crt\""
Sep 11 12:04:20.472903 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.472907    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cbc380\"/\"openshift-service-ca.crt\""
Sep 11 12:04:20.473077 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.472927    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cbc380\"/\"default-dockercfg-r7zjq\""
Sep 11 12:04:20.473249 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.473234    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cbc380\"/\"test-model-cbc380-kserve-self-signed-certs\""
Sep 11 12:04:20.492167 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.492145    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"]
Sep 11 12:04:20.531165 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.531140    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-dshm\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.531269 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.531190    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-tmp-dir\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.531269 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.531216    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-home\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.531269 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.531232    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60eb5415-3c27-4862-b861-c660face056c-tls-certs\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.531269 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.531268    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-model-cache\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.531436 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.531366    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvwxd\" (UniqueName: \"kubernetes.io/projected/60eb5415-3c27-4862-b861-c660face056c-kube-api-access-bvwxd\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.632673 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.632644    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-tmp-dir\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.632820 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.632678    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-home\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.632820 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.632702    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60eb5415-3c27-4862-b861-c660face056c-tls-certs\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.632820 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.632743    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-model-cache\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.632820 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.632790    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvwxd\" (UniqueName: \"kubernetes.io/projected/60eb5415-3c27-4862-b861-c660face056c-kube-api-access-bvwxd\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.633022 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.632830    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-dshm\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.633147 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.633073    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-home\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.633147 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.633127    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-tmp-dir\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.633249 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.633169    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-model-cache\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.635082 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.635062    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-dshm\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.635402 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.635384    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60eb5415-3c27-4862-b861-c660face056c-tls-certs\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.642721 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.642700    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvwxd\" (UniqueName: \"kubernetes.io/projected/60eb5415-3c27-4862-b861-c660face056c-kube-api-access-bvwxd\") pod \"test-model-cbc380-kserve-6f59ffbf5-qc8vd\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") " pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:20.773965 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:20.773939    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:21.091161 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:21.091134    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"]
Sep 11 12:04:21.091854 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:04:21.091827    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60eb5415_3c27_4862_b861_c660face056c.slice/crio-23d8d7fec0dd5291592b1e3edab6960a3148f56887e48296846d295b0ff2703a WatchSource:0}: Error finding container 23d8d7fec0dd5291592b1e3edab6960a3148f56887e48296846d295b0ff2703a: Status 404 returned error can't find the container with id 23d8d7fec0dd5291592b1e3edab6960a3148f56887e48296846d295b0ff2703a
Sep 11 12:04:21.268761 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:21.268728    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"]
Sep 11 12:04:21.268983 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:21.268958    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c" podUID="b1accd26-cf3c-4538-8773-bb9ac5090e3f" containerName="payload-processing" containerID="cri-o://529b16a49516ff8459f3e854edbf268e908537ae48f9b04a52cdfa9c186f595e" gracePeriod=30
Sep 11 12:04:21.513980 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:21.513952    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"
Sep 11 12:04:21.643750 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:21.643724    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xl9mq\" (UniqueName: \"kubernetes.io/projected/b1accd26-cf3c-4538-8773-bb9ac5090e3f-kube-api-access-xl9mq\") pod \"b1accd26-cf3c-4538-8773-bb9ac5090e3f\" (UID: \"b1accd26-cf3c-4538-8773-bb9ac5090e3f\") "
Sep 11 12:04:21.643974 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:21.643952    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1accd26-cf3c-4538-8773-bb9ac5090e3f-plugins-config-volume\") pod \"b1accd26-cf3c-4538-8773-bb9ac5090e3f\" (UID: \"b1accd26-cf3c-4538-8773-bb9ac5090e3f\") "
Sep 11 12:04:21.644470 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:21.644439    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1accd26-cf3c-4538-8773-bb9ac5090e3f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b1accd26-cf3c-4538-8773-bb9ac5090e3f" (UID: "b1accd26-cf3c-4538-8773-bb9ac5090e3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:04:21.646068 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:21.646040    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1accd26-cf3c-4538-8773-bb9ac5090e3f-kube-api-access-xl9mq" (OuterVolumeSpecName: "kube-api-access-xl9mq") pod "b1accd26-cf3c-4538-8773-bb9ac5090e3f" (UID: "b1accd26-cf3c-4538-8773-bb9ac5090e3f"). InnerVolumeSpecName "kube-api-access-xl9mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:04:21.689302 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:21.689277    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-cdec04-rcrw4"]
Sep 11 12:04:21.693446 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:21.693420    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-cdec04-rcrw4"]
Sep 11 12:04:21.745375 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:21.745354    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl9mq\" (UniqueName: \"kubernetes.io/projected/b1accd26-cf3c-4538-8773-bb9ac5090e3f-kube-api-access-xl9mq\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:21.745375 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:21.745377    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1accd26-cf3c-4538-8773-bb9ac5090e3f-plugins-config-volume\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:22.072822 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:22.072782    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4e40b1c-872e-4148-a904-2c32c99613f2" path="/var/lib/kubelet/pods/d4e40b1c-872e-4148-a904-2c32c99613f2/volumes"
Sep 11 12:04:22.073392 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:22.073364    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd" event={"ID":"60eb5415-3c27-4862-b861-c660face056c","Type":"ContainerStarted","Data":"42159ed3f603a3fa6511ace35479d323b394edae7f9e057a12964fccfdb9e652"}
Sep 11 12:04:22.073526 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:22.073457    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd" event={"ID":"60eb5415-3c27-4862-b861-c660face056c","Type":"ContainerStarted","Data":"23d8d7fec0dd5291592b1e3edab6960a3148f56887e48296846d295b0ff2703a"}
Sep 11 12:04:22.074396 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:22.074373    2569 generic.go:356] "Generic (PLEG): container finished" podID="b1accd26-cf3c-4538-8773-bb9ac5090e3f" containerID="529b16a49516ff8459f3e854edbf268e908537ae48f9b04a52cdfa9c186f595e" exitCode=0
Sep 11 12:04:22.074501 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:22.074408    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"
Sep 11 12:04:22.074501 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:22.074423    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c" event={"ID":"b1accd26-cf3c-4538-8773-bb9ac5090e3f","Type":"ContainerDied","Data":"529b16a49516ff8459f3e854edbf268e908537ae48f9b04a52cdfa9c186f595e"}
Sep 11 12:04:22.074501 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:22.074459    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c" event={"ID":"b1accd26-cf3c-4538-8773-bb9ac5090e3f","Type":"ContainerDied","Data":"c7da48023413af0a766b5151256863143dd9fc30df05b1a5aa9ce58f0197ab76"}
Sep 11 12:04:22.074501 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:22.074482    2569 scope.go:117] "RemoveContainer" containerID="529b16a49516ff8459f3e854edbf268e908537ae48f9b04a52cdfa9c186f595e"
Sep 11 12:04:22.084040 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:22.084021    2569 scope.go:117] "RemoveContainer" containerID="529b16a49516ff8459f3e854edbf268e908537ae48f9b04a52cdfa9c186f595e"
Sep 11 12:04:22.084358 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:04:22.084338    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529b16a49516ff8459f3e854edbf268e908537ae48f9b04a52cdfa9c186f595e\": container with ID starting with 529b16a49516ff8459f3e854edbf268e908537ae48f9b04a52cdfa9c186f595e not found: ID does not exist" containerID="529b16a49516ff8459f3e854edbf268e908537ae48f9b04a52cdfa9c186f595e"
Sep 11 12:04:22.084428 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:22.084365    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529b16a49516ff8459f3e854edbf268e908537ae48f9b04a52cdfa9c186f595e"} err="failed to get container status \"529b16a49516ff8459f3e854edbf268e908537ae48f9b04a52cdfa9c186f595e\": rpc error: code = NotFound desc = could not find container \"529b16a49516ff8459f3e854edbf268e908537ae48f9b04a52cdfa9c186f595e\": container with ID starting with 529b16a49516ff8459f3e854edbf268e908537ae48f9b04a52cdfa9c186f595e not found: ID does not exist"
Sep 11 12:04:22.097293 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:22.097243    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd" podStartSLOduration=2.09722624 podStartE2EDuration="2.09722624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:04:22.092843668 +0000 UTC m=+984.514712067" watchObservedRunningTime="2026-09-11 12:04:22.09722624 +0000 UTC m=+984.519094622"
Sep 11 12:04:22.114800 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:22.114772    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"]
Sep 11 12:04:22.117282 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:22.117262    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-cdec04-5cf4975955-bhc9c"]
Sep 11 12:04:24.070033 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:24.070002    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1accd26-cf3c-4538-8773-bb9ac5090e3f" path="/var/lib/kubelet/pods/b1accd26-cf3c-4538-8773-bb9ac5090e3f/volumes"
Sep 11 12:04:24.853535 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:24.853497    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"]
Sep 11 12:04:24.853948 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:24.853935    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1accd26-cf3c-4538-8773-bb9ac5090e3f" containerName="payload-processing"
Sep 11 12:04:24.854015 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:24.853950    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1accd26-cf3c-4538-8773-bb9ac5090e3f" containerName="payload-processing"
Sep 11 12:04:24.854061 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:24.854053    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1accd26-cf3c-4538-8773-bb9ac5090e3f" containerName="payload-processing"
Sep 11 12:04:24.856965 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:24.856946    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:04:24.859550 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:24.859526    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0eb76ba5-serving-cert\""
Sep 11 12:04:24.860100 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:24.860086    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 12:04:24.860233 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:24.860213    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rrxjh\""
Sep 11 12:04:24.875306 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:24.875285    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"]
Sep 11 12:04:24.973086 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:24.973053    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-openshift-service-ca\") pod \"maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn\" (UID: \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:04:24.973241 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:24.973117    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkb76\" (UniqueName: \"kubernetes.io/projected/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-kube-api-access-qkb76\") pod \"maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn\" (UID: \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:04:24.973241 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:24.973139    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-maas-api-tls\") pod \"maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn\" (UID: \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:04:24.973241 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:24.973201    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-metrics-certs\") pod \"maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn\" (UID: \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:04:25.073694 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:25.073660    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-openshift-service-ca\") pod \"maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn\" (UID: \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:04:25.074120 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:25.073828    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkb76\" (UniqueName: \"kubernetes.io/projected/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-kube-api-access-qkb76\") pod \"maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn\" (UID: \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:04:25.074120 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:25.073873    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-maas-api-tls\") pod \"maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn\" (UID: \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:04:25.074120 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:25.073908    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-metrics-certs\") pod \"maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn\" (UID: \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:04:25.074447 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:25.074418    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-openshift-service-ca\") pod \"maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn\" (UID: \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:04:25.076396 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:25.076373    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-maas-api-tls\") pod \"maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn\" (UID: \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:04:25.076486 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:25.076436    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-metrics-certs\") pod \"maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn\" (UID: \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:04:25.092821 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:25.092794    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkb76\" (UniqueName: \"kubernetes.io/projected/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-kube-api-access-qkb76\") pod \"maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn\" (UID: \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:04:25.167135 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:25.167074    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:04:25.444995 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:25.444971    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"]
Sep 11 12:04:25.445449 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:04:25.445424    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb4dc21_6152_4ae4_8fc8_9ab903f9e6f4.slice/crio-b1b8962ced334af0ea226c5251e33ca47bb10abe7de0a32b799b6fc6590a36c4 WatchSource:0}: Error finding container b1b8962ced334af0ea226c5251e33ca47bb10abe7de0a32b799b6fc6590a36c4: Status 404 returned error can't find the container with id b1b8962ced334af0ea226c5251e33ca47bb10abe7de0a32b799b6fc6590a36c4
Sep 11 12:04:25.446991 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:25.446973    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:04:25.541484 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:25.541447    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"]
Sep 11 12:04:26.098933 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:26.098902    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn" event={"ID":"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4","Type":"ContainerStarted","Data":"b1b8962ced334af0ea226c5251e33ca47bb10abe7de0a32b799b6fc6590a36c4"}
Sep 11 12:04:27.107022 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:27.106983    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn" event={"ID":"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4","Type":"ContainerStarted","Data":"7d561a8108bba5fe4b12d3b06017f2c75f3da1f38e37d3e1bbc1036d443e95d8"}
Sep 11 12:04:27.107469 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:27.107132    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:04:27.139187 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:27.139124    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn" podStartSLOduration=2.080537033 podStartE2EDuration="3.139103409s" totalImagesPullingTime="1.058566376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:04:27.136050111 +0000 UTC m=+989.557918492" watchObservedRunningTime="2026-09-11 12:04:27.139103409 +0000 UTC m=+989.560971834"
Sep 11 12:04:30.774389 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:30.774354    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:30.774815 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:30.774409    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:30.786926 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:30.786905    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:31.143197 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:31.143123    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:04:33.119466 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:33.119438    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:04:36.486565 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:36.486479    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"]
Sep 11 12:04:40.560580 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:40.560546    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n" podUID="1fff0481-aad2-4ee7-b82f-5bd37efc7ffc" containerName="main" containerID="cri-o://fada7176c68abf21d4cb18937e930c3dc2f925f89041b91859ff061e57c2d9d4" gracePeriod=45
Sep 11 12:04:40.805473 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:40.805450    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:04:40.918073 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:40.917990    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-model-cache\") pod \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") "
Sep 11 12:04:40.918073 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:40.918032    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c2cq\" (UniqueName: \"kubernetes.io/projected/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-kube-api-access-6c2cq\") pod \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") "
Sep 11 12:04:40.918073 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:40.918060    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-dshm\") pod \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") "
Sep 11 12:04:40.918349 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:40.918083    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-tls-certs\") pod \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") "
Sep 11 12:04:40.918349 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:40.918146    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-home\") pod \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") "
Sep 11 12:04:40.918349 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:40.918289    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-tmp-dir\") pod \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\" (UID: \"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc\") "
Sep 11 12:04:40.918349 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:40.918318    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-model-cache" (OuterVolumeSpecName: "model-cache") pod "1fff0481-aad2-4ee7-b82f-5bd37efc7ffc" (UID: "1fff0481-aad2-4ee7-b82f-5bd37efc7ffc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:04:40.918539 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:40.918406    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-home" (OuterVolumeSpecName: "home") pod "1fff0481-aad2-4ee7-b82f-5bd37efc7ffc" (UID: "1fff0481-aad2-4ee7-b82f-5bd37efc7ffc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:04:40.918539 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:40.918516    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1fff0481-aad2-4ee7-b82f-5bd37efc7ffc" (UID: "1fff0481-aad2-4ee7-b82f-5bd37efc7ffc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:04:40.918723 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:40.918703    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-model-cache\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:40.918723 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:40.918723    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-home\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:40.918867 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:40.918735    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-tmp-dir\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:40.920254 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:40.920224    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-dshm" (OuterVolumeSpecName: "dshm") pod "1fff0481-aad2-4ee7-b82f-5bd37efc7ffc" (UID: "1fff0481-aad2-4ee7-b82f-5bd37efc7ffc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:04:40.920352 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:40.920300    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-kube-api-access-6c2cq" (OuterVolumeSpecName: "kube-api-access-6c2cq") pod "1fff0481-aad2-4ee7-b82f-5bd37efc7ffc" (UID: "1fff0481-aad2-4ee7-b82f-5bd37efc7ffc"). InnerVolumeSpecName "kube-api-access-6c2cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:04:40.920352 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:40.920301    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1fff0481-aad2-4ee7-b82f-5bd37efc7ffc" (UID: "1fff0481-aad2-4ee7-b82f-5bd37efc7ffc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:04:41.019294 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:41.019262    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c2cq\" (UniqueName: \"kubernetes.io/projected/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-kube-api-access-6c2cq\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:41.019294 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:41.019292    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-dshm\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:41.019413 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:41.019303    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc-tls-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:41.192372 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:41.192313    2569 generic.go:356] "Generic (PLEG): container finished" podID="1fff0481-aad2-4ee7-b82f-5bd37efc7ffc" containerID="fada7176c68abf21d4cb18937e930c3dc2f925f89041b91859ff061e57c2d9d4" exitCode=0
Sep 11 12:04:41.192372 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:41.192355    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"
Sep 11 12:04:41.192508 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:41.192397    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n" event={"ID":"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc","Type":"ContainerDied","Data":"fada7176c68abf21d4cb18937e930c3dc2f925f89041b91859ff061e57c2d9d4"}
Sep 11 12:04:41.192508 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:41.192431    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n" event={"ID":"1fff0481-aad2-4ee7-b82f-5bd37efc7ffc","Type":"ContainerDied","Data":"21d342f3f9041517844394327af20cf492cb8b13c38362746f4b128eb96b3fcf"}
Sep 11 12:04:41.192508 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:41.192452    2569 scope.go:117] "RemoveContainer" containerID="fada7176c68abf21d4cb18937e930c3dc2f925f89041b91859ff061e57c2d9d4"
Sep 11 12:04:41.202069 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:41.202054    2569 scope.go:117] "RemoveContainer" containerID="fada7176c68abf21d4cb18937e930c3dc2f925f89041b91859ff061e57c2d9d4"
Sep 11 12:04:41.202307 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:04:41.202292    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fada7176c68abf21d4cb18937e930c3dc2f925f89041b91859ff061e57c2d9d4\": container with ID starting with fada7176c68abf21d4cb18937e930c3dc2f925f89041b91859ff061e57c2d9d4 not found: ID does not exist" containerID="fada7176c68abf21d4cb18937e930c3dc2f925f89041b91859ff061e57c2d9d4"
Sep 11 12:04:41.202356 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:41.202312    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fada7176c68abf21d4cb18937e930c3dc2f925f89041b91859ff061e57c2d9d4"} err="failed to get container status \"fada7176c68abf21d4cb18937e930c3dc2f925f89041b91859ff061e57c2d9d4\": rpc error: code = NotFound desc = could not find container \"fada7176c68abf21d4cb18937e930c3dc2f925f89041b91859ff061e57c2d9d4\": container with ID starting with fada7176c68abf21d4cb18937e930c3dc2f925f89041b91859ff061e57c2d9d4 not found: ID does not exist"
Sep 11 12:04:41.219777 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:41.219750    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"]
Sep 11 12:04:41.223935 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:41.223913    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-cdec04/autores-model-cdec04-kserve-654cc5f6f9-t7f8n"]
Sep 11 12:04:42.070086 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:42.070053    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fff0481-aad2-4ee7-b82f-5bd37efc7ffc" path="/var/lib/kubelet/pods/1fff0481-aad2-4ee7-b82f-5bd37efc7ffc/volumes"
Sep 11 12:04:51.505471 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:51.505406    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd" podUID="a9c52e16-56a8-424f-bf94-5eebff6c1963" containerName="main" containerID="cri-o://12b0b10b683c93b2a07d7e8c243f26c4fdfabe9deec50a5b597e30cb2a785374" gracePeriod=45
Sep 11 12:04:51.752850 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:51.752827    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:51.923452 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:51.923372    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-model-cache\") pod \"a9c52e16-56a8-424f-bf94-5eebff6c1963\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") "
Sep 11 12:04:51.923452 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:51.923449    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-dshm\") pod \"a9c52e16-56a8-424f-bf94-5eebff6c1963\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") "
Sep 11 12:04:51.923704 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:51.923486    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9c52e16-56a8-424f-bf94-5eebff6c1963-tls-certs\") pod \"a9c52e16-56a8-424f-bf94-5eebff6c1963\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") "
Sep 11 12:04:51.923704 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:51.923517    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-tmp-dir\") pod \"a9c52e16-56a8-424f-bf94-5eebff6c1963\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") "
Sep 11 12:04:51.923704 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:51.923538    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-home\") pod \"a9c52e16-56a8-424f-bf94-5eebff6c1963\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") "
Sep 11 12:04:51.923704 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:51.923641    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mr28\" (UniqueName: \"kubernetes.io/projected/a9c52e16-56a8-424f-bf94-5eebff6c1963-kube-api-access-4mr28\") pod \"a9c52e16-56a8-424f-bf94-5eebff6c1963\" (UID: \"a9c52e16-56a8-424f-bf94-5eebff6c1963\") "
Sep 11 12:04:51.923889 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:51.923694    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-model-cache" (OuterVolumeSpecName: "model-cache") pod "a9c52e16-56a8-424f-bf94-5eebff6c1963" (UID: "a9c52e16-56a8-424f-bf94-5eebff6c1963"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:04:51.923889 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:51.923833    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a9c52e16-56a8-424f-bf94-5eebff6c1963" (UID: "a9c52e16-56a8-424f-bf94-5eebff6c1963"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:04:51.923889 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:51.923858    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-home" (OuterVolumeSpecName: "home") pod "a9c52e16-56a8-424f-bf94-5eebff6c1963" (UID: "a9c52e16-56a8-424f-bf94-5eebff6c1963"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:04:51.924057 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:51.924007    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-tmp-dir\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:51.924057 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:51.924021    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-home\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:51.924057 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:51.924030    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-model-cache\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:51.925764 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:51.925742    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c52e16-56a8-424f-bf94-5eebff6c1963-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9c52e16-56a8-424f-bf94-5eebff6c1963" (UID: "a9c52e16-56a8-424f-bf94-5eebff6c1963"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:04:51.925872 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:51.925766    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-dshm" (OuterVolumeSpecName: "dshm") pod "a9c52e16-56a8-424f-bf94-5eebff6c1963" (UID: "a9c52e16-56a8-424f-bf94-5eebff6c1963"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:04:51.925872 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:51.925849    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c52e16-56a8-424f-bf94-5eebff6c1963-kube-api-access-4mr28" (OuterVolumeSpecName: "kube-api-access-4mr28") pod "a9c52e16-56a8-424f-bf94-5eebff6c1963" (UID: "a9c52e16-56a8-424f-bf94-5eebff6c1963"). InnerVolumeSpecName "kube-api-access-4mr28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:04:52.025216 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:52.025189    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9c52e16-56a8-424f-bf94-5eebff6c1963-tls-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:52.025216 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:52.025212    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mr28\" (UniqueName: \"kubernetes.io/projected/a9c52e16-56a8-424f-bf94-5eebff6c1963-kube-api-access-4mr28\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:52.025351 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:52.025222    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9c52e16-56a8-424f-bf94-5eebff6c1963-dshm\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:52.258249 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:52.258220    2569 generic.go:356] "Generic (PLEG): container finished" podID="a9c52e16-56a8-424f-bf94-5eebff6c1963" containerID="12b0b10b683c93b2a07d7e8c243f26c4fdfabe9deec50a5b597e30cb2a785374" exitCode=0
Sep 11 12:04:52.258405 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:52.258261    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"
Sep 11 12:04:52.258405 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:52.258304    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd" event={"ID":"a9c52e16-56a8-424f-bf94-5eebff6c1963","Type":"ContainerDied","Data":"12b0b10b683c93b2a07d7e8c243f26c4fdfabe9deec50a5b597e30cb2a785374"}
Sep 11 12:04:52.258405 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:52.258350    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd" event={"ID":"a9c52e16-56a8-424f-bf94-5eebff6c1963","Type":"ContainerDied","Data":"03a3e8407578ff5c424dc3f0ac96a164237a0f890bffd2a22c62b2198884f3cf"}
Sep 11 12:04:52.258405 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:52.258371    2569 scope.go:117] "RemoveContainer" containerID="12b0b10b683c93b2a07d7e8c243f26c4fdfabe9deec50a5b597e30cb2a785374"
Sep 11 12:04:52.267282 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:52.267256    2569 scope.go:117] "RemoveContainer" containerID="12b0b10b683c93b2a07d7e8c243f26c4fdfabe9deec50a5b597e30cb2a785374"
Sep 11 12:04:52.267565 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:04:52.267544    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b0b10b683c93b2a07d7e8c243f26c4fdfabe9deec50a5b597e30cb2a785374\": container with ID starting with 12b0b10b683c93b2a07d7e8c243f26c4fdfabe9deec50a5b597e30cb2a785374 not found: ID does not exist" containerID="12b0b10b683c93b2a07d7e8c243f26c4fdfabe9deec50a5b597e30cb2a785374"
Sep 11 12:04:52.267629 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:52.267570    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b0b10b683c93b2a07d7e8c243f26c4fdfabe9deec50a5b597e30cb2a785374"} err="failed to get container status \"12b0b10b683c93b2a07d7e8c243f26c4fdfabe9deec50a5b597e30cb2a785374\": rpc error: code = NotFound desc = could not find container \"12b0b10b683c93b2a07d7e8c243f26c4fdfabe9deec50a5b597e30cb2a785374\": container with ID starting with 12b0b10b683c93b2a07d7e8c243f26c4fdfabe9deec50a5b597e30cb2a785374 not found: ID does not exist"
Sep 11 12:04:52.277727 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:52.277707    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"]
Sep 11 12:04:52.282701 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:52.282683    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-f67d8993-kserve-6bf5d58dc9-w22fd"]
Sep 11 12:04:53.570925 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:53.570888    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf"]
Sep 11 12:04:53.571389 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:53.571282    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9c52e16-56a8-424f-bf94-5eebff6c1963" containerName="main"
Sep 11 12:04:53.571389 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:53.571294    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9c52e16-56a8-424f-bf94-5eebff6c1963" containerName="main"
Sep 11 12:04:53.571389 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:53.571312    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fff0481-aad2-4ee7-b82f-5bd37efc7ffc" containerName="main"
Sep 11 12:04:53.571389 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:53.571318    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fff0481-aad2-4ee7-b82f-5bd37efc7ffc" containerName="main"
Sep 11 12:04:53.571389 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:53.571375    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fff0481-aad2-4ee7-b82f-5bd37efc7ffc" containerName="main"
Sep 11 12:04:53.571389 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:53.571384    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9c52e16-56a8-424f-bf94-5eebff6c1963" containerName="main"
Sep 11 12:04:53.575796 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:53.575776    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf"
Sep 11 12:04:53.579858 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:53.579834    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xkpf\""
Sep 11 12:04:53.586418 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:53.586395    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf"]
Sep 11 12:04:53.637365 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:53.637334    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2dc4364-ff33-4e80-877b-017a22e1878e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf\" (UID: \"e2dc4364-ff33-4e80-877b-017a22e1878e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf"
Sep 11 12:04:53.737839 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:53.737811    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2dc4364-ff33-4e80-877b-017a22e1878e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf\" (UID: \"e2dc4364-ff33-4e80-877b-017a22e1878e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf"
Sep 11 12:04:53.738504 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:53.738485    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2dc4364-ff33-4e80-877b-017a22e1878e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf\" (UID: \"e2dc4364-ff33-4e80-877b-017a22e1878e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf"
Sep 11 12:04:53.887304 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:53.887229    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf"
Sep 11 12:04:54.069795 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:54.069762    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9c52e16-56a8-424f-bf94-5eebff6c1963" path="/var/lib/kubelet/pods/a9c52e16-56a8-424f-bf94-5eebff6c1963/volumes"
Sep 11 12:04:54.160320 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:54.160252    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf"]
Sep 11 12:04:54.164299 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:04:54.164268    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2dc4364_ff33_4e80_877b_017a22e1878e.slice/crio-a87abf99ca9f827e0d0c93c89b1dd18fe476a8a69e9f2b37c1a91793155a3688 WatchSource:0}: Error finding container a87abf99ca9f827e0d0c93c89b1dd18fe476a8a69e9f2b37c1a91793155a3688: Status 404 returned error can't find the container with id a87abf99ca9f827e0d0c93c89b1dd18fe476a8a69e9f2b37c1a91793155a3688
Sep 11 12:04:54.272531 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:54.272506    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf" event={"ID":"e2dc4364-ff33-4e80-877b-017a22e1878e","Type":"ContainerStarted","Data":"a87abf99ca9f827e0d0c93c89b1dd18fe476a8a69e9f2b37c1a91793155a3688"}
Sep 11 12:04:55.280552 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:55.280515    2569 generic.go:356] "Generic (PLEG): container finished" podID="e2dc4364-ff33-4e80-877b-017a22e1878e" containerID="131e6d89c702fc0e23509c18fcf301e47987885abf040b7eec9f92696c992ca6" exitCode=0
Sep 11 12:04:55.280921 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:55.280551    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf" event={"ID":"e2dc4364-ff33-4e80-877b-017a22e1878e","Type":"ContainerDied","Data":"131e6d89c702fc0e23509c18fcf301e47987885abf040b7eec9f92696c992ca6"}
Sep 11 12:04:56.416062 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:56.416036    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf"
Sep 11 12:04:56.464829 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:56.464803    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2dc4364-ff33-4e80-877b-017a22e1878e-maas-api-service-ca\") pod \"e2dc4364-ff33-4e80-877b-017a22e1878e\" (UID: \"e2dc4364-ff33-4e80-877b-017a22e1878e\") "
Sep 11 12:04:56.465237 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:56.465211    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2dc4364-ff33-4e80-877b-017a22e1878e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e2dc4364-ff33-4e80-877b-017a22e1878e" (UID: "e2dc4364-ff33-4e80-877b-017a22e1878e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:04:56.565697 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:56.565641    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2dc4364-ff33-4e80-877b-017a22e1878e-maas-api-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:04:57.297826 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:57.297788    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf" event={"ID":"e2dc4364-ff33-4e80-877b-017a22e1878e","Type":"ContainerDied","Data":"a87abf99ca9f827e0d0c93c89b1dd18fe476a8a69e9f2b37c1a91793155a3688"}
Sep 11 12:04:57.298008 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:57.297848    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a87abf99ca9f827e0d0c93c89b1dd18fe476a8a69e9f2b37c1a91793155a3688"
Sep 11 12:04:57.298008 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:57.297799    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf"
Sep 11 12:04:58.018836 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:04:58.018805    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"]
Sep 11 12:05:05.887628 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:05.887597    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"]
Sep 11 12:05:05.888099 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:05.887800    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn" podUID="1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4" containerName="maas-api" containerID="cri-o://7d561a8108bba5fe4b12d3b06017f2c75f3da1f38e37d3e1bbc1036d443e95d8" gracePeriod=30
Sep 11 12:05:06.142356 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.142297    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:05:06.232937 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.232909    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf"]
Sep 11 12:05:06.242610 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.238141    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb76ba5-x8rtf"]
Sep 11 12:05:06.249486 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.249458    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkb76\" (UniqueName: \"kubernetes.io/projected/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-kube-api-access-qkb76\") pod \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\" (UID: \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\") "
Sep 11 12:05:06.249618 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.249524    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-metrics-certs\") pod \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\" (UID: \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\") "
Sep 11 12:05:06.249618 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.249569    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-openshift-service-ca\") pod \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\" (UID: \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\") "
Sep 11 12:05:06.249737 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.249712    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-maas-api-tls\") pod \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\" (UID: \"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4\") "
Sep 11 12:05:06.249979 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.249953    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4" (UID: "1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:05:06.251754 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.251733    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4" (UID: "1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:05:06.252201 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.252180    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4" (UID: "1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:05:06.252201 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.252193    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-kube-api-access-qkb76" (OuterVolumeSpecName: "kube-api-access-qkb76") pod "1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4" (UID: "1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4"). InnerVolumeSpecName "kube-api-access-qkb76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:05:06.350809 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.350777    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkb76\" (UniqueName: \"kubernetes.io/projected/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-kube-api-access-qkb76\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:05:06.350809 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.350809    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-metrics-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:05:06.351052 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.350825    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-openshift-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:05:06.351052 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.350838    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4-maas-api-tls\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:05:06.357011 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.356978    2569 generic.go:356] "Generic (PLEG): container finished" podID="1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4" containerID="7d561a8108bba5fe4b12d3b06017f2c75f3da1f38e37d3e1bbc1036d443e95d8" exitCode=0
Sep 11 12:05:06.357163 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.357041    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"
Sep 11 12:05:06.357163 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.357059    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn" event={"ID":"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4","Type":"ContainerDied","Data":"7d561a8108bba5fe4b12d3b06017f2c75f3da1f38e37d3e1bbc1036d443e95d8"}
Sep 11 12:05:06.357163 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.357149    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn" event={"ID":"1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4","Type":"ContainerDied","Data":"b1b8962ced334af0ea226c5251e33ca47bb10abe7de0a32b799b6fc6590a36c4"}
Sep 11 12:05:06.357286 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.357179    2569 scope.go:117] "RemoveContainer" containerID="7d561a8108bba5fe4b12d3b06017f2c75f3da1f38e37d3e1bbc1036d443e95d8"
Sep 11 12:05:06.370077 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.370060    2569 scope.go:117] "RemoveContainer" containerID="7d561a8108bba5fe4b12d3b06017f2c75f3da1f38e37d3e1bbc1036d443e95d8"
Sep 11 12:05:06.370371 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:05:06.370347    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d561a8108bba5fe4b12d3b06017f2c75f3da1f38e37d3e1bbc1036d443e95d8\": container with ID starting with 7d561a8108bba5fe4b12d3b06017f2c75f3da1f38e37d3e1bbc1036d443e95d8 not found: ID does not exist" containerID="7d561a8108bba5fe4b12d3b06017f2c75f3da1f38e37d3e1bbc1036d443e95d8"
Sep 11 12:05:06.370427 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.370384    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d561a8108bba5fe4b12d3b06017f2c75f3da1f38e37d3e1bbc1036d443e95d8"} err="failed to get container status \"7d561a8108bba5fe4b12d3b06017f2c75f3da1f38e37d3e1bbc1036d443e95d8\": rpc error: code = NotFound desc = could not find container \"7d561a8108bba5fe4b12d3b06017f2c75f3da1f38e37d3e1bbc1036d443e95d8\": container with ID starting with 7d561a8108bba5fe4b12d3b06017f2c75f3da1f38e37d3e1bbc1036d443e95d8 not found: ID does not exist"
Sep 11 12:05:06.389713 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.389691    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"]
Sep 11 12:05:06.405873 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:06.405811    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0eb76ba5-79d8f47d9b-frgsn"]
Sep 11 12:05:08.070464 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:08.070423    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4" path="/var/lib/kubelet/pods/1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4/volumes"
Sep 11 12:05:08.070963 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:08.070932    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2dc4364-ff33-4e80-877b-017a22e1878e" path="/var/lib/kubelet/pods/e2dc4364-ff33-4e80-877b-017a22e1878e/volumes"
Sep 11 12:05:10.501263 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.501232    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"]
Sep 11 12:05:10.501839 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.501818    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2dc4364-ff33-4e80-877b-017a22e1878e" containerName="revoke-keys"
Sep 11 12:05:10.501906 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.501844    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2dc4364-ff33-4e80-877b-017a22e1878e" containerName="revoke-keys"
Sep 11 12:05:10.501906 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.501895    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4" containerName="maas-api"
Sep 11 12:05:10.501906 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.501903    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4" containerName="maas-api"
Sep 11 12:05:10.502032 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.501996    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2dc4364-ff33-4e80-877b-017a22e1878e" containerName="revoke-keys"
Sep 11 12:05:10.502032 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.502005    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eb4dc21-6152-4ae4-8fc8-9ab903f9e6f4" containerName="maas-api"
Sep 11 12:05:10.505975 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.505958    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:05:10.512041 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.512022    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 12:05:10.515240 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.515120    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-69b1ea-serving-cert\""
Sep 11 12:05:10.521766 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.521736    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rrxjh\""
Sep 11 12:05:10.523729 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.523710    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"]
Sep 11 12:05:10.591267 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.591243    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mr9p\" (UniqueName: \"kubernetes.io/projected/9a071285-1513-442a-9f80-576e28b29e39-kube-api-access-2mr9p\") pod \"maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s\" (UID: \"9a071285-1513-442a-9f80-576e28b29e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:05:10.591389 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.591298    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a071285-1513-442a-9f80-576e28b29e39-maas-api-tls\") pod \"maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s\" (UID: \"9a071285-1513-442a-9f80-576e28b29e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:05:10.591389 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.591366    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a071285-1513-442a-9f80-576e28b29e39-metrics-certs\") pod \"maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s\" (UID: \"9a071285-1513-442a-9f80-576e28b29e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:05:10.591467 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.591390    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a071285-1513-442a-9f80-576e28b29e39-openshift-service-ca\") pod \"maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s\" (UID: \"9a071285-1513-442a-9f80-576e28b29e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:05:10.692332 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.692301    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a071285-1513-442a-9f80-576e28b29e39-metrics-certs\") pod \"maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s\" (UID: \"9a071285-1513-442a-9f80-576e28b29e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:05:10.692466 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.692344    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a071285-1513-442a-9f80-576e28b29e39-openshift-service-ca\") pod \"maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s\" (UID: \"9a071285-1513-442a-9f80-576e28b29e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:05:10.692466 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.692415    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mr9p\" (UniqueName: \"kubernetes.io/projected/9a071285-1513-442a-9f80-576e28b29e39-kube-api-access-2mr9p\") pod \"maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s\" (UID: \"9a071285-1513-442a-9f80-576e28b29e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:05:10.692607 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.692482    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a071285-1513-442a-9f80-576e28b29e39-maas-api-tls\") pod \"maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s\" (UID: \"9a071285-1513-442a-9f80-576e28b29e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:05:10.693034 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.693013    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a071285-1513-442a-9f80-576e28b29e39-openshift-service-ca\") pod \"maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s\" (UID: \"9a071285-1513-442a-9f80-576e28b29e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:05:10.694860 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.694839    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a071285-1513-442a-9f80-576e28b29e39-metrics-certs\") pod \"maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s\" (UID: \"9a071285-1513-442a-9f80-576e28b29e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:05:10.694990 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.694879    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a071285-1513-442a-9f80-576e28b29e39-maas-api-tls\") pod \"maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s\" (UID: \"9a071285-1513-442a-9f80-576e28b29e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:05:10.709198 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.709180    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mr9p\" (UniqueName: \"kubernetes.io/projected/9a071285-1513-442a-9f80-576e28b29e39-kube-api-access-2mr9p\") pod \"maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s\" (UID: \"9a071285-1513-442a-9f80-576e28b29e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:05:10.818740 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:10.818682    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:05:11.135885 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:11.135854    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"]
Sep 11 12:05:11.136973 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:05:11.136945    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a071285_1513_442a_9f80_576e28b29e39.slice/crio-7d8744c61fa19a4c50deeab5a1328455a2cebc3cee592eae224a51f11caf003f WatchSource:0}: Error finding container 7d8744c61fa19a4c50deeab5a1328455a2cebc3cee592eae224a51f11caf003f: Status 404 returned error can't find the container with id 7d8744c61fa19a4c50deeab5a1328455a2cebc3cee592eae224a51f11caf003f
Sep 11 12:05:11.391120 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:11.391032    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s" event={"ID":"9a071285-1513-442a-9f80-576e28b29e39","Type":"ContainerStarted","Data":"7d8744c61fa19a4c50deeab5a1328455a2cebc3cee592eae224a51f11caf003f"}
Sep 11 12:05:12.399257 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:12.399223    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s" event={"ID":"9a071285-1513-442a-9f80-576e28b29e39","Type":"ContainerStarted","Data":"745fc53c7429ba8e439351849f78db7dd14cd42b34afaea150bbbedcfa406216"}
Sep 11 12:05:12.399749 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:12.399286    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:05:12.430439 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:12.430388    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s" podStartSLOduration=1.997563655 podStartE2EDuration="2.430373369s" totalImagesPullingTime="432.809714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:05:12.427877907 +0000 UTC m=+1034.849746286" watchObservedRunningTime="2026-09-11 12:05:12.430373369 +0000 UTC m=+1034.852241748"
Sep 11 12:05:13.037607 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.037556    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg" podUID="7b482ce5-cda8-4696-a3e0-ec196382c0e1" containerName="main" containerID="cri-o://94f9e6a32ec2396549c805429470d5987d5e434f289e83431413bfa7d6046a17" gracePeriod=45
Sep 11 12:05:13.291085 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.291026    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:05:13.407476 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.407441    2569 generic.go:356] "Generic (PLEG): container finished" podID="7b482ce5-cda8-4696-a3e0-ec196382c0e1" containerID="94f9e6a32ec2396549c805429470d5987d5e434f289e83431413bfa7d6046a17" exitCode=0
Sep 11 12:05:13.407476 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.407475    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"
Sep 11 12:05:13.407891 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.407526    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg" event={"ID":"7b482ce5-cda8-4696-a3e0-ec196382c0e1","Type":"ContainerDied","Data":"94f9e6a32ec2396549c805429470d5987d5e434f289e83431413bfa7d6046a17"}
Sep 11 12:05:13.407891 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.407563    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg" event={"ID":"7b482ce5-cda8-4696-a3e0-ec196382c0e1","Type":"ContainerDied","Data":"31a44f9de58465ded1e24cc3ca21a7234f7f83eec4e36a6a77f3b784e8cff1b9"}
Sep 11 12:05:13.407891 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.407598    2569 scope.go:117] "RemoveContainer" containerID="94f9e6a32ec2396549c805429470d5987d5e434f289e83431413bfa7d6046a17"
Sep 11 12:05:13.416363 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.416345    2569 scope.go:117] "RemoveContainer" containerID="94f9e6a32ec2396549c805429470d5987d5e434f289e83431413bfa7d6046a17"
Sep 11 12:05:13.416648 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:05:13.416631    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94f9e6a32ec2396549c805429470d5987d5e434f289e83431413bfa7d6046a17\": container with ID starting with 94f9e6a32ec2396549c805429470d5987d5e434f289e83431413bfa7d6046a17 not found: ID does not exist" containerID="94f9e6a32ec2396549c805429470d5987d5e434f289e83431413bfa7d6046a17"
Sep 11 12:05:13.416695 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.416658    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94f9e6a32ec2396549c805429470d5987d5e434f289e83431413bfa7d6046a17"} err="failed to get container status \"94f9e6a32ec2396549c805429470d5987d5e434f289e83431413bfa7d6046a17\": rpc error: code = NotFound desc = could not find container \"94f9e6a32ec2396549c805429470d5987d5e434f289e83431413bfa7d6046a17\": container with ID starting with 94f9e6a32ec2396549c805429470d5987d5e434f289e83431413bfa7d6046a17 not found: ID does not exist"
Sep 11 12:05:13.417869 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.417854    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b482ce5-cda8-4696-a3e0-ec196382c0e1-tls-certs\") pod \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") "
Sep 11 12:05:13.417913 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.417893    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-model-cache\") pod \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") "
Sep 11 12:05:13.417960 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.417919    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wdw6\" (UniqueName: \"kubernetes.io/projected/7b482ce5-cda8-4696-a3e0-ec196382c0e1-kube-api-access-5wdw6\") pod \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") "
Sep 11 12:05:13.418019 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.417988    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-home\") pod \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") "
Sep 11 12:05:13.418072 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.418023    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-dshm\") pod \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") "
Sep 11 12:05:13.418127 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.418110    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-tmp-dir\") pod \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\" (UID: \"7b482ce5-cda8-4696-a3e0-ec196382c0e1\") "
Sep 11 12:05:13.418183 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.418165    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b482ce5-cda8-4696-a3e0-ec196382c0e1" (UID: "7b482ce5-cda8-4696-a3e0-ec196382c0e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:05:13.418298 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.418269    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-home" (OuterVolumeSpecName: "home") pod "7b482ce5-cda8-4696-a3e0-ec196382c0e1" (UID: "7b482ce5-cda8-4696-a3e0-ec196382c0e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:05:13.418451 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.418432    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b482ce5-cda8-4696-a3e0-ec196382c0e1" (UID: "7b482ce5-cda8-4696-a3e0-ec196382c0e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:05:13.418505 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.418436    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-model-cache\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:05:13.418505 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.418469    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-home\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:05:13.419971 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.419952    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-dshm" (OuterVolumeSpecName: "dshm") pod "7b482ce5-cda8-4696-a3e0-ec196382c0e1" (UID: "7b482ce5-cda8-4696-a3e0-ec196382c0e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:05:13.420421 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.420394    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b482ce5-cda8-4696-a3e0-ec196382c0e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b482ce5-cda8-4696-a3e0-ec196382c0e1" (UID: "7b482ce5-cda8-4696-a3e0-ec196382c0e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:05:13.420503 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.420398    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b482ce5-cda8-4696-a3e0-ec196382c0e1-kube-api-access-5wdw6" (OuterVolumeSpecName: "kube-api-access-5wdw6") pod "7b482ce5-cda8-4696-a3e0-ec196382c0e1" (UID: "7b482ce5-cda8-4696-a3e0-ec196382c0e1"). InnerVolumeSpecName "kube-api-access-5wdw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:05:13.519776 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.519754    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-tmp-dir\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:05:13.519776 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.519774    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b482ce5-cda8-4696-a3e0-ec196382c0e1-tls-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:05:13.519907 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.519783    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wdw6\" (UniqueName: \"kubernetes.io/projected/7b482ce5-cda8-4696-a3e0-ec196382c0e1-kube-api-access-5wdw6\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:05:13.519907 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.519793    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b482ce5-cda8-4696-a3e0-ec196382c0e1-dshm\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:05:13.742225 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.742194    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"]
Sep 11 12:05:13.747184 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:13.747157    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-f67d8993-kserve-6ddb968877-qczvg"]
Sep 11 12:05:14.071332 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:14.071289    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b482ce5-cda8-4696-a3e0-ec196382c0e1" path="/var/lib/kubelet/pods/7b482ce5-cda8-4696-a3e0-ec196382c0e1/volumes"
Sep 11 12:05:18.413752 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:18.413723    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:05:29.951390 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:29.951357    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"]
Sep 11 12:05:29.951875 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:29.951779    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b482ce5-cda8-4696-a3e0-ec196382c0e1" containerName="main"
Sep 11 12:05:29.951875 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:29.951792    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b482ce5-cda8-4696-a3e0-ec196382c0e1" containerName="main"
Sep 11 12:05:29.951875 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:29.951864    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b482ce5-cda8-4696-a3e0-ec196382c0e1" containerName="main"
Sep 11 12:05:29.954787 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:29.954770    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:29.957924 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:29.957902    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-12d2a9\"/\"default-dockercfg-jnccg\""
Sep 11 12:05:29.958218 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:29.958201    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-12d2a9\"/\"openshift-service-ca.crt\""
Sep 11 12:05:29.959032 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:29.959014    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-12d2a9\"/\"test-model-12d2a9-kserve-self-signed-certs\""
Sep 11 12:05:29.959190 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:29.959172    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-12d2a9\"/\"kube-root-ca.crt\""
Sep 11 12:05:29.966933 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:29.966911    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"]
Sep 11 12:05:30.061517 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.061494    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cbpj\" (UniqueName: \"kubernetes.io/projected/d108905c-bf96-46fa-b9a4-875bdc47cb74-kube-api-access-7cbpj\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.061657 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.061536    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-dshm\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.061657 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.061564    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-home\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.061657 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.061644    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-model-cache\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.061790 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.061671    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d108905c-bf96-46fa-b9a4-875bdc47cb74-tls-certs\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.061790 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.061699    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-tmp-dir\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.162846 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.162822    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cbpj\" (UniqueName: \"kubernetes.io/projected/d108905c-bf96-46fa-b9a4-875bdc47cb74-kube-api-access-7cbpj\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.162972 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.162868    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-dshm\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.162972 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.162888    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-home\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.162972 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.162926    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-model-cache\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.162972 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.162957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d108905c-bf96-46fa-b9a4-875bdc47cb74-tls-certs\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.163180 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.163014    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-tmp-dir\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.163314 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.163290    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-home\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.163459 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.163439    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-model-cache\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.163581 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.163560    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-tmp-dir\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.165258 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.165236    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-dshm\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.165414 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.165397    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d108905c-bf96-46fa-b9a4-875bdc47cb74-tls-certs\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.172909 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.172888    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cbpj\" (UniqueName: \"kubernetes.io/projected/d108905c-bf96-46fa-b9a4-875bdc47cb74-kube-api-access-7cbpj\") pod \"test-model-12d2a9-kserve-5d546b98d7-dkd6h\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") " pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.264243 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.264217    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:30.541767 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:30.541741    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"]
Sep 11 12:05:30.544098 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:05:30.544069    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd108905c_bf96_46fa_b9a4_875bdc47cb74.slice/crio-95b5b56b5927f4df6fcb14d2bd9661c8fd187d665aedbd80315c034df9cb4687 WatchSource:0}: Error finding container 95b5b56b5927f4df6fcb14d2bd9661c8fd187d665aedbd80315c034df9cb4687: Status 404 returned error can't find the container with id 95b5b56b5927f4df6fcb14d2bd9661c8fd187d665aedbd80315c034df9cb4687
Sep 11 12:05:31.517239 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:31.517204    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h" event={"ID":"d108905c-bf96-46fa-b9a4-875bdc47cb74","Type":"ContainerStarted","Data":"909270c1bed2c2caee9ebf9e053f0dbbd319ae93984845fed723cd4de86fec22"}
Sep 11 12:05:31.517638 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:31.517248    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h" event={"ID":"d108905c-bf96-46fa-b9a4-875bdc47cb74","Type":"ContainerStarted","Data":"95b5b56b5927f4df6fcb14d2bd9661c8fd187d665aedbd80315c034df9cb4687"}
Sep 11 12:05:31.539171 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:31.539119    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h" podStartSLOduration=2.539105627 podStartE2EDuration="2.539105627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:05:31.536102836 +0000 UTC m=+1053.957971503" watchObservedRunningTime="2026-09-11 12:05:31.539105627 +0000 UTC m=+1053.960974011"
Sep 11 12:05:40.264785 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:40.264754    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:40.265272 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:40.265049    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:40.277151 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:40.277129    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:40.582810 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:40.582738    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:05:48.250675 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.250633    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"]
Sep 11 12:05:48.257195 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.257161    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.262601 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.262472    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-69b1ea\"/\"kube-root-ca.crt\""
Sep 11 12:05:48.263040 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.263017    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-69b1ea\"/\"explicit-model-69b1ea-kserve-self-signed-certs\""
Sep 11 12:05:48.264845 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.263977    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-69b1ea\"/\"default-dockercfg-sbzkq\""
Sep 11 12:05:48.264964 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.264033    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-69b1ea\"/\"openshift-service-ca.crt\""
Sep 11 12:05:48.267215 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.267189    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"]
Sep 11 12:05:48.318901 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.318879    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9bf5\" (UniqueName: \"kubernetes.io/projected/8f33fc3e-0055-415d-95ec-10c462ef8561-kube-api-access-f9bf5\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.319022 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.318911    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-model-cache\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.319022 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.318928    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f33fc3e-0055-415d-95ec-10c462ef8561-tls-certs\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.319022 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.318955    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-dshm\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.319132 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.319050    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-tmp-dir\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.319132 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.319096    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-home\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.420168 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.420137    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-tmp-dir\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.420336 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.420205    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-home\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.420336 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.420284    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9bf5\" (UniqueName: \"kubernetes.io/projected/8f33fc3e-0055-415d-95ec-10c462ef8561-kube-api-access-f9bf5\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.420336 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.420316    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-model-cache\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.420511 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.420341    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f33fc3e-0055-415d-95ec-10c462ef8561-tls-certs\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.420511 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.420375    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-dshm\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.420632 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.420525    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-tmp-dir\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.420698 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.420626    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-home\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.420698 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.420636    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-model-cache\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.422472 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.422447    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-dshm\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.422799 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.422782    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f33fc3e-0055-415d-95ec-10c462ef8561-tls-certs\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.428329 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.428309    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9bf5\" (UniqueName: \"kubernetes.io/projected/8f33fc3e-0055-415d-95ec-10c462ef8561-kube-api-access-f9bf5\") pod \"explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") " pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.572039 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.571967    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:05:48.850898 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:48.850872    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"]
Sep 11 12:05:48.851560 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:05:48.851529    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f33fc3e_0055_415d_95ec_10c462ef8561.slice/crio-ea945e5f64eb6bc3aa6290a233110ba3a5b6e06f602100d0b700c7badcd6d2ba WatchSource:0}: Error finding container ea945e5f64eb6bc3aa6290a233110ba3a5b6e06f602100d0b700c7badcd6d2ba: Status 404 returned error can't find the container with id ea945e5f64eb6bc3aa6290a233110ba3a5b6e06f602100d0b700c7badcd6d2ba
Sep 11 12:05:49.626435 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:49.626404    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv" event={"ID":"8f33fc3e-0055-415d-95ec-10c462ef8561","Type":"ContainerStarted","Data":"a5e7886554bbb406c32317dd982c6bde0841fc6423c4099308068e9be682a3fb"}
Sep 11 12:05:49.626832 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:49.626446    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv" event={"ID":"8f33fc3e-0055-415d-95ec-10c462ef8561","Type":"ContainerStarted","Data":"ea945e5f64eb6bc3aa6290a233110ba3a5b6e06f602100d0b700c7badcd6d2ba"}
Sep 11 12:05:49.660094 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:49.660039    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv" podStartSLOduration=1.660024295 podStartE2EDuration="1.660024295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:05:49.658301531 +0000 UTC m=+1072.080169930" watchObservedRunningTime="2026-09-11 12:05:49.660024295 +0000 UTC m=+1072.081892675"
Sep 11 12:05:50.915686 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:50.915610    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"]
Sep 11 12:05:58.572745 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:05:58.572715    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:06:01.657375 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:01.657346    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"]
Sep 11 12:06:01.657866 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:01.657552    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s" podUID="9a071285-1513-442a-9f80-576e28b29e39" containerName="maas-api" containerID="cri-o://745fc53c7429ba8e439351849f78db7dd14cd42b34afaea150bbbedcfa406216" gracePeriod=30
Sep 11 12:06:01.913850 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:01.913802    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:06:02.045378 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.045350    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a071285-1513-442a-9f80-576e28b29e39-metrics-certs\") pod \"9a071285-1513-442a-9f80-576e28b29e39\" (UID: \"9a071285-1513-442a-9f80-576e28b29e39\") "
Sep 11 12:06:02.045530 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.045395    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mr9p\" (UniqueName: \"kubernetes.io/projected/9a071285-1513-442a-9f80-576e28b29e39-kube-api-access-2mr9p\") pod \"9a071285-1513-442a-9f80-576e28b29e39\" (UID: \"9a071285-1513-442a-9f80-576e28b29e39\") "
Sep 11 12:06:02.045530 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.045412    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a071285-1513-442a-9f80-576e28b29e39-maas-api-tls\") pod \"9a071285-1513-442a-9f80-576e28b29e39\" (UID: \"9a071285-1513-442a-9f80-576e28b29e39\") "
Sep 11 12:06:02.045530 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.045488    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a071285-1513-442a-9f80-576e28b29e39-openshift-service-ca\") pod \"9a071285-1513-442a-9f80-576e28b29e39\" (UID: \"9a071285-1513-442a-9f80-576e28b29e39\") "
Sep 11 12:06:02.045905 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.045881    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a071285-1513-442a-9f80-576e28b29e39-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9a071285-1513-442a-9f80-576e28b29e39" (UID: "9a071285-1513-442a-9f80-576e28b29e39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:06:02.047464 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.047435    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a071285-1513-442a-9f80-576e28b29e39-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9a071285-1513-442a-9f80-576e28b29e39" (UID: "9a071285-1513-442a-9f80-576e28b29e39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:06:02.047566 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.047482    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a071285-1513-442a-9f80-576e28b29e39-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9a071285-1513-442a-9f80-576e28b29e39" (UID: "9a071285-1513-442a-9f80-576e28b29e39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:06:02.047566 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.047517    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a071285-1513-442a-9f80-576e28b29e39-kube-api-access-2mr9p" (OuterVolumeSpecName: "kube-api-access-2mr9p") pod "9a071285-1513-442a-9f80-576e28b29e39" (UID: "9a071285-1513-442a-9f80-576e28b29e39"). InnerVolumeSpecName "kube-api-access-2mr9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:06:02.146426 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.146399    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a071285-1513-442a-9f80-576e28b29e39-metrics-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:02.146426 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.146421    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mr9p\" (UniqueName: \"kubernetes.io/projected/9a071285-1513-442a-9f80-576e28b29e39-kube-api-access-2mr9p\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:02.146426 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.146430    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a071285-1513-442a-9f80-576e28b29e39-maas-api-tls\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:02.146638 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.146438    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a071285-1513-442a-9f80-576e28b29e39-openshift-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:02.704167 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.704123    2569 generic.go:356] "Generic (PLEG): container finished" podID="9a071285-1513-442a-9f80-576e28b29e39" containerID="745fc53c7429ba8e439351849f78db7dd14cd42b34afaea150bbbedcfa406216" exitCode=0
Sep 11 12:06:02.704669 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.704190    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"
Sep 11 12:06:02.704669 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.704208    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s" event={"ID":"9a071285-1513-442a-9f80-576e28b29e39","Type":"ContainerDied","Data":"745fc53c7429ba8e439351849f78db7dd14cd42b34afaea150bbbedcfa406216"}
Sep 11 12:06:02.704669 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.704310    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s" event={"ID":"9a071285-1513-442a-9f80-576e28b29e39","Type":"ContainerDied","Data":"7d8744c61fa19a4c50deeab5a1328455a2cebc3cee592eae224a51f11caf003f"}
Sep 11 12:06:02.704669 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.704341    2569 scope.go:117] "RemoveContainer" containerID="745fc53c7429ba8e439351849f78db7dd14cd42b34afaea150bbbedcfa406216"
Sep 11 12:06:02.714321 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.714308    2569 scope.go:117] "RemoveContainer" containerID="745fc53c7429ba8e439351849f78db7dd14cd42b34afaea150bbbedcfa406216"
Sep 11 12:06:02.714609 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:06:02.714574    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"745fc53c7429ba8e439351849f78db7dd14cd42b34afaea150bbbedcfa406216\": container with ID starting with 745fc53c7429ba8e439351849f78db7dd14cd42b34afaea150bbbedcfa406216 not found: ID does not exist" containerID="745fc53c7429ba8e439351849f78db7dd14cd42b34afaea150bbbedcfa406216"
Sep 11 12:06:02.714691 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.714615    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745fc53c7429ba8e439351849f78db7dd14cd42b34afaea150bbbedcfa406216"} err="failed to get container status \"745fc53c7429ba8e439351849f78db7dd14cd42b34afaea150bbbedcfa406216\": rpc error: code = NotFound desc = could not find container \"745fc53c7429ba8e439351849f78db7dd14cd42b34afaea150bbbedcfa406216\": container with ID starting with 745fc53c7429ba8e439351849f78db7dd14cd42b34afaea150bbbedcfa406216 not found: ID does not exist"
Sep 11 12:06:02.724558 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.724534    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"]
Sep 11 12:06:02.727084 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:02.727060    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-69b1ea-658b658c9-jdr4s"]
Sep 11 12:06:04.071416 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:04.071386    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a071285-1513-442a-9f80-576e28b29e39" path="/var/lib/kubelet/pods/9a071285-1513-442a-9f80-576e28b29e39/volumes"
Sep 11 12:06:06.657401 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:06.657363    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv" podUID="8f33fc3e-0055-415d-95ec-10c462ef8561" containerName="main" containerID="cri-o://a5e7886554bbb406c32317dd982c6bde0841fc6423c4099308068e9be682a3fb" gracePeriod=45
Sep 11 12:06:06.906107 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:06.906083    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:06:06.990765 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:06.990690    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f33fc3e-0055-415d-95ec-10c462ef8561-tls-certs\") pod \"8f33fc3e-0055-415d-95ec-10c462ef8561\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") "
Sep 11 12:06:06.990765 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:06.990726    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-tmp-dir\") pod \"8f33fc3e-0055-415d-95ec-10c462ef8561\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") "
Sep 11 12:06:06.990967 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:06.990852    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-dshm\") pod \"8f33fc3e-0055-415d-95ec-10c462ef8561\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") "
Sep 11 12:06:06.990967 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:06.990903    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-home\") pod \"8f33fc3e-0055-415d-95ec-10c462ef8561\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") "
Sep 11 12:06:06.990967 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:06.990924    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-model-cache\") pod \"8f33fc3e-0055-415d-95ec-10c462ef8561\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") "
Sep 11 12:06:06.990967 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:06.990950    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f33fc3e-0055-415d-95ec-10c462ef8561" (UID: "8f33fc3e-0055-415d-95ec-10c462ef8561"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:06:06.991176 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:06.990975    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9bf5\" (UniqueName: \"kubernetes.io/projected/8f33fc3e-0055-415d-95ec-10c462ef8561-kube-api-access-f9bf5\") pod \"8f33fc3e-0055-415d-95ec-10c462ef8561\" (UID: \"8f33fc3e-0055-415d-95ec-10c462ef8561\") "
Sep 11 12:06:06.991176 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:06.991148    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-home" (OuterVolumeSpecName: "home") pod "8f33fc3e-0055-415d-95ec-10c462ef8561" (UID: "8f33fc3e-0055-415d-95ec-10c462ef8561"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:06:06.991176 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:06.991162    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f33fc3e-0055-415d-95ec-10c462ef8561" (UID: "8f33fc3e-0055-415d-95ec-10c462ef8561"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:06:06.991392 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:06.991330    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-home\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:06.991392 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:06.991341    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-model-cache\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:06.991392 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:06.991350    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-tmp-dir\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:06.993013 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:06.992992    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-dshm" (OuterVolumeSpecName: "dshm") pod "8f33fc3e-0055-415d-95ec-10c462ef8561" (UID: "8f33fc3e-0055-415d-95ec-10c462ef8561"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:06:06.993013 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:06.993000    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f33fc3e-0055-415d-95ec-10c462ef8561-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f33fc3e-0055-415d-95ec-10c462ef8561" (UID: "8f33fc3e-0055-415d-95ec-10c462ef8561"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:06:06.993147 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:06.993037    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f33fc3e-0055-415d-95ec-10c462ef8561-kube-api-access-f9bf5" (OuterVolumeSpecName: "kube-api-access-f9bf5") pod "8f33fc3e-0055-415d-95ec-10c462ef8561" (UID: "8f33fc3e-0055-415d-95ec-10c462ef8561"). InnerVolumeSpecName "kube-api-access-f9bf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:06:07.092690 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:07.092667    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f33fc3e-0055-415d-95ec-10c462ef8561-tls-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:07.092690 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:07.092687    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f33fc3e-0055-415d-95ec-10c462ef8561-dshm\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:07.092833 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:07.092696    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9bf5\" (UniqueName: \"kubernetes.io/projected/8f33fc3e-0055-415d-95ec-10c462ef8561-kube-api-access-f9bf5\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:07.735984 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:07.735952    2569 generic.go:356] "Generic (PLEG): container finished" podID="8f33fc3e-0055-415d-95ec-10c462ef8561" containerID="a5e7886554bbb406c32317dd982c6bde0841fc6423c4099308068e9be682a3fb" exitCode=0
Sep 11 12:06:07.736407 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:07.735991    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"
Sep 11 12:06:07.736407 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:07.736037    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv" event={"ID":"8f33fc3e-0055-415d-95ec-10c462ef8561","Type":"ContainerDied","Data":"a5e7886554bbb406c32317dd982c6bde0841fc6423c4099308068e9be682a3fb"}
Sep 11 12:06:07.736407 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:07.736075    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv" event={"ID":"8f33fc3e-0055-415d-95ec-10c462ef8561","Type":"ContainerDied","Data":"ea945e5f64eb6bc3aa6290a233110ba3a5b6e06f602100d0b700c7badcd6d2ba"}
Sep 11 12:06:07.736407 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:07.736093    2569 scope.go:117] "RemoveContainer" containerID="a5e7886554bbb406c32317dd982c6bde0841fc6423c4099308068e9be682a3fb"
Sep 11 12:06:07.746749 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:07.746733    2569 scope.go:117] "RemoveContainer" containerID="a5e7886554bbb406c32317dd982c6bde0841fc6423c4099308068e9be682a3fb"
Sep 11 12:06:07.747007 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:06:07.746989    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e7886554bbb406c32317dd982c6bde0841fc6423c4099308068e9be682a3fb\": container with ID starting with a5e7886554bbb406c32317dd982c6bde0841fc6423c4099308068e9be682a3fb not found: ID does not exist" containerID="a5e7886554bbb406c32317dd982c6bde0841fc6423c4099308068e9be682a3fb"
Sep 11 12:06:07.747098 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:07.747014    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e7886554bbb406c32317dd982c6bde0841fc6423c4099308068e9be682a3fb"} err="failed to get container status \"a5e7886554bbb406c32317dd982c6bde0841fc6423c4099308068e9be682a3fb\": rpc error: code = NotFound desc = could not find container \"a5e7886554bbb406c32317dd982c6bde0841fc6423c4099308068e9be682a3fb\": container with ID starting with a5e7886554bbb406c32317dd982c6bde0841fc6423c4099308068e9be682a3fb not found: ID does not exist"
Sep 11 12:06:07.758638 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:07.758616    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"]
Sep 11 12:06:07.762858 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:07.762838    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-69b1ea/explicit-model-69b1ea-kserve-5cb9bd86f-rc7wv"]
Sep 11 12:06:08.080369 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:08.080337    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f33fc3e-0055-415d-95ec-10c462ef8561" path="/var/lib/kubelet/pods/8f33fc3e-0055-415d-95ec-10c462ef8561/volumes"
Sep 11 12:06:20.336991 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.336963    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"]
Sep 11 12:06:20.337415 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.337337    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f33fc3e-0055-415d-95ec-10c462ef8561" containerName="main"
Sep 11 12:06:20.337415 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.337348    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f33fc3e-0055-415d-95ec-10c462ef8561" containerName="main"
Sep 11 12:06:20.337415 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.337365    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a071285-1513-442a-9f80-576e28b29e39" containerName="maas-api"
Sep 11 12:06:20.337415 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.337371    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a071285-1513-442a-9f80-576e28b29e39" containerName="maas-api"
Sep 11 12:06:20.337549 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.337432    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a071285-1513-442a-9f80-576e28b29e39" containerName="maas-api"
Sep 11 12:06:20.337549 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.337442    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f33fc3e-0055-415d-95ec-10c462ef8561" containerName="main"
Sep 11 12:06:20.341955 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.341937    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.344863 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.344839    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4a674b\"/\"default-dockercfg-wvqlv\""
Sep 11 12:06:20.345749 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.345725    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4a674b\"/\"kube-root-ca.crt\""
Sep 11 12:06:20.345749 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.345744    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4a674b\"/\"noten-model-4a674b-kserve-self-signed-certs\""
Sep 11 12:06:20.345904 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.345769    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4a674b\"/\"openshift-service-ca.crt\""
Sep 11 12:06:20.353343 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.353317    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"]
Sep 11 12:06:20.515129 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.515097    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-home\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.515350 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.515144    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e979668d-1a6c-41f2-90cd-5e3037482c3d-tls-certs\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.515350 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.515170    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcnf6\" (UniqueName: \"kubernetes.io/projected/e979668d-1a6c-41f2-90cd-5e3037482c3d-kube-api-access-xcnf6\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.515350 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.515226    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-dshm\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.515350 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.515250    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-tmp-dir\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.515350 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.515324    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-model-cache\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.616098 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.616024    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-model-cache\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.616098 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.616074    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-home\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.616098 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.616097    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e979668d-1a6c-41f2-90cd-5e3037482c3d-tls-certs\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.616297 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.616114    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcnf6\" (UniqueName: \"kubernetes.io/projected/e979668d-1a6c-41f2-90cd-5e3037482c3d-kube-api-access-xcnf6\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.616297 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.616141    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-dshm\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.616297 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.616160    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-tmp-dir\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.616449 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.616427    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-model-cache\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.616495 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.616456    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-home\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.616540 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.616528    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-tmp-dir\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.618439 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.618415    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-dshm\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.618728 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.618709    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e979668d-1a6c-41f2-90cd-5e3037482c3d-tls-certs\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.626071 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.626053    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcnf6\" (UniqueName: \"kubernetes.io/projected/e979668d-1a6c-41f2-90cd-5e3037482c3d-kube-api-access-xcnf6\") pod \"noten-model-4a674b-kserve-69cc5dcbff-fhcg5\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") " pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.651970 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.651946    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:20.924645 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:20.924614    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"]
Sep 11 12:06:20.925643 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:06:20.925616    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode979668d_1a6c_41f2_90cd_5e3037482c3d.slice/crio-56b2c19c662099f0dc2c80a5d28ef3e5b0049f8e68ed4dc239436a6f951057d9 WatchSource:0}: Error finding container 56b2c19c662099f0dc2c80a5d28ef3e5b0049f8e68ed4dc239436a6f951057d9: Status 404 returned error can't find the container with id 56b2c19c662099f0dc2c80a5d28ef3e5b0049f8e68ed4dc239436a6f951057d9
Sep 11 12:06:21.821663 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:21.821625    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5" event={"ID":"e979668d-1a6c-41f2-90cd-5e3037482c3d","Type":"ContainerStarted","Data":"7491868dc02d2d6884646cfc816f8153bf1c5375f2d2ba9e7034894b4e12509d"}
Sep 11 12:06:21.822014 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:21.821674    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5" event={"ID":"e979668d-1a6c-41f2-90cd-5e3037482c3d","Type":"ContainerStarted","Data":"56b2c19c662099f0dc2c80a5d28ef3e5b0049f8e68ed4dc239436a6f951057d9"}
Sep 11 12:06:21.842686 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:21.842644    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5" podStartSLOduration=1.842629664 podStartE2EDuration="1.842629664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:06:21.840505271 +0000 UTC m=+1104.262373671" watchObservedRunningTime="2026-09-11 12:06:21.842629664 +0000 UTC m=+1104.264498044"
Sep 11 12:06:30.652164 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:30.652129    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:30.652556 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:30.652182    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:30.664830 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:30.664805    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:30.891426 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:30.891396    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:33.155258 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:33.155225    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"]
Sep 11 12:06:38.202826 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:38.202784    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"]
Sep 11 12:06:39.225683 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:39.225651    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"]
Sep 11 12:06:39.350844 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:39.350789    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cbc380-h758j"]
Sep 11 12:06:39.355084 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:39.355063    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cbc380-h758j"
Sep 11 12:06:39.358025 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:39.358005    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xkpf\""
Sep 11 12:06:39.368088 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:39.366016    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cbc380-h758j"]
Sep 11 12:06:39.371916 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:39.371892    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9543c7f-5985-4d9a-a5c5-ad8a72415894-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cbc380-h758j\" (UID: \"e9543c7f-5985-4d9a-a5c5-ad8a72415894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cbc380-h758j"
Sep 11 12:06:39.472661 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:39.472623    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9543c7f-5985-4d9a-a5c5-ad8a72415894-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cbc380-h758j\" (UID: \"e9543c7f-5985-4d9a-a5c5-ad8a72415894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cbc380-h758j"
Sep 11 12:06:39.473254 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:39.473233    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9543c7f-5985-4d9a-a5c5-ad8a72415894-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cbc380-h758j\" (UID: \"e9543c7f-5985-4d9a-a5c5-ad8a72415894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cbc380-h758j"
Sep 11 12:06:39.668983 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:39.668946    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cbc380-h758j"
Sep 11 12:06:39.944905 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:39.944865    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cbc380-h758j"]
Sep 11 12:06:39.946059 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:06:39.946040    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9543c7f_5985_4d9a_a5c5_ad8a72415894.slice/crio-911b28a0db5644ef375b6600e93018b41615b52b6c0c51d6f2ef0c6cdb8490cf WatchSource:0}: Error finding container 911b28a0db5644ef375b6600e93018b41615b52b6c0c51d6f2ef0c6cdb8490cf: Status 404 returned error can't find the container with id 911b28a0db5644ef375b6600e93018b41615b52b6c0c51d6f2ef0c6cdb8490cf
Sep 11 12:06:40.934838 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:40.934761    2569 generic.go:356] "Generic (PLEG): container finished" podID="e9543c7f-5985-4d9a-a5c5-ad8a72415894" containerID="ffe61f3074e5bb2e2a980aa16b70d3620cc023c68134b86cd2a9a985dca8a460" exitCode=0
Sep 11 12:06:40.935164 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:40.934844    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cbc380-h758j" event={"ID":"e9543c7f-5985-4d9a-a5c5-ad8a72415894","Type":"ContainerDied","Data":"ffe61f3074e5bb2e2a980aa16b70d3620cc023c68134b86cd2a9a985dca8a460"}
Sep 11 12:06:40.935164 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:40.934919    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cbc380-h758j" event={"ID":"e9543c7f-5985-4d9a-a5c5-ad8a72415894","Type":"ContainerStarted","Data":"911b28a0db5644ef375b6600e93018b41615b52b6c0c51d6f2ef0c6cdb8490cf"}
Sep 11 12:06:42.068235 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:42.068209    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cbc380-h758j"
Sep 11 12:06:42.098969 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:42.098941    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9543c7f-5985-4d9a-a5c5-ad8a72415894-maas-api-service-ca\") pod \"e9543c7f-5985-4d9a-a5c5-ad8a72415894\" (UID: \"e9543c7f-5985-4d9a-a5c5-ad8a72415894\") "
Sep 11 12:06:42.099251 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:42.099233    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9543c7f-5985-4d9a-a5c5-ad8a72415894-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9543c7f-5985-4d9a-a5c5-ad8a72415894" (UID: "e9543c7f-5985-4d9a-a5c5-ad8a72415894"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:06:42.099344 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:42.099330    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9543c7f-5985-4d9a-a5c5-ad8a72415894-maas-api-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:42.947820 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:42.947788    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cbc380-h758j"
Sep 11 12:06:42.947987 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:42.947822    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cbc380-h758j" event={"ID":"e9543c7f-5985-4d9a-a5c5-ad8a72415894","Type":"ContainerDied","Data":"911b28a0db5644ef375b6600e93018b41615b52b6c0c51d6f2ef0c6cdb8490cf"}
Sep 11 12:06:42.947987 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:42.947872    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="911b28a0db5644ef375b6600e93018b41615b52b6c0c51d6f2ef0c6cdb8490cf"
Sep 11 12:06:48.174545 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.174489    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5" podUID="e979668d-1a6c-41f2-90cd-5e3037482c3d" containerName="main" containerID="cri-o://7491868dc02d2d6884646cfc816f8153bf1c5375f2d2ba9e7034894b4e12509d" gracePeriod=45
Sep 11 12:06:48.424395 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.424367    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:48.559780 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.559754    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-model-cache\") pod \"e979668d-1a6c-41f2-90cd-5e3037482c3d\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") "
Sep 11 12:06:48.559967 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.559790    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcnf6\" (UniqueName: \"kubernetes.io/projected/e979668d-1a6c-41f2-90cd-5e3037482c3d-kube-api-access-xcnf6\") pod \"e979668d-1a6c-41f2-90cd-5e3037482c3d\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") "
Sep 11 12:06:48.559967 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.559810    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-home\") pod \"e979668d-1a6c-41f2-90cd-5e3037482c3d\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") "
Sep 11 12:06:48.559967 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.559853    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e979668d-1a6c-41f2-90cd-5e3037482c3d-tls-certs\") pod \"e979668d-1a6c-41f2-90cd-5e3037482c3d\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") "
Sep 11 12:06:48.559967 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.559905    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-dshm\") pod \"e979668d-1a6c-41f2-90cd-5e3037482c3d\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") "
Sep 11 12:06:48.560173 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.560011    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-tmp-dir\") pod \"e979668d-1a6c-41f2-90cd-5e3037482c3d\" (UID: \"e979668d-1a6c-41f2-90cd-5e3037482c3d\") "
Sep 11 12:06:48.560173 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.560034    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e979668d-1a6c-41f2-90cd-5e3037482c3d" (UID: "e979668d-1a6c-41f2-90cd-5e3037482c3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:06:48.560173 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.560132    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-home" (OuterVolumeSpecName: "home") pod "e979668d-1a6c-41f2-90cd-5e3037482c3d" (UID: "e979668d-1a6c-41f2-90cd-5e3037482c3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:06:48.560323 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.560297    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e979668d-1a6c-41f2-90cd-5e3037482c3d" (UID: "e979668d-1a6c-41f2-90cd-5e3037482c3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:06:48.560392 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.560378    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-model-cache\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:48.560453 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.560398    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-home\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:48.560453 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.560430    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-tmp-dir\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:48.562059 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.562026    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-dshm" (OuterVolumeSpecName: "dshm") pod "e979668d-1a6c-41f2-90cd-5e3037482c3d" (UID: "e979668d-1a6c-41f2-90cd-5e3037482c3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:06:48.562168 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.562052    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e979668d-1a6c-41f2-90cd-5e3037482c3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e979668d-1a6c-41f2-90cd-5e3037482c3d" (UID: "e979668d-1a6c-41f2-90cd-5e3037482c3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:06:48.562168 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.562100    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e979668d-1a6c-41f2-90cd-5e3037482c3d-kube-api-access-xcnf6" (OuterVolumeSpecName: "kube-api-access-xcnf6") pod "e979668d-1a6c-41f2-90cd-5e3037482c3d" (UID: "e979668d-1a6c-41f2-90cd-5e3037482c3d"). InnerVolumeSpecName "kube-api-access-xcnf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:06:48.661309 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.661285    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e979668d-1a6c-41f2-90cd-5e3037482c3d-tls-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:48.661309 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.661305    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e979668d-1a6c-41f2-90cd-5e3037482c3d-dshm\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:48.661453 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.661315    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcnf6\" (UniqueName: \"kubernetes.io/projected/e979668d-1a6c-41f2-90cd-5e3037482c3d-kube-api-access-xcnf6\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:48.984413 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.984323    2569 generic.go:356] "Generic (PLEG): container finished" podID="e979668d-1a6c-41f2-90cd-5e3037482c3d" containerID="7491868dc02d2d6884646cfc816f8153bf1c5375f2d2ba9e7034894b4e12509d" exitCode=0
Sep 11 12:06:48.984413 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.984365    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"
Sep 11 12:06:48.984413 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.984406    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5" event={"ID":"e979668d-1a6c-41f2-90cd-5e3037482c3d","Type":"ContainerDied","Data":"7491868dc02d2d6884646cfc816f8153bf1c5375f2d2ba9e7034894b4e12509d"}
Sep 11 12:06:48.984757 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.984448    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5" event={"ID":"e979668d-1a6c-41f2-90cd-5e3037482c3d","Type":"ContainerDied","Data":"56b2c19c662099f0dc2c80a5d28ef3e5b0049f8e68ed4dc239436a6f951057d9"}
Sep 11 12:06:48.984757 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.984470    2569 scope.go:117] "RemoveContainer" containerID="7491868dc02d2d6884646cfc816f8153bf1c5375f2d2ba9e7034894b4e12509d"
Sep 11 12:06:48.995001 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.994981    2569 scope.go:117] "RemoveContainer" containerID="7491868dc02d2d6884646cfc816f8153bf1c5375f2d2ba9e7034894b4e12509d"
Sep 11 12:06:48.995259 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:06:48.995237    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7491868dc02d2d6884646cfc816f8153bf1c5375f2d2ba9e7034894b4e12509d\": container with ID starting with 7491868dc02d2d6884646cfc816f8153bf1c5375f2d2ba9e7034894b4e12509d not found: ID does not exist" containerID="7491868dc02d2d6884646cfc816f8153bf1c5375f2d2ba9e7034894b4e12509d"
Sep 11 12:06:48.995329 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:48.995267    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7491868dc02d2d6884646cfc816f8153bf1c5375f2d2ba9e7034894b4e12509d"} err="failed to get container status \"7491868dc02d2d6884646cfc816f8153bf1c5375f2d2ba9e7034894b4e12509d\": rpc error: code = NotFound desc = could not find container \"7491868dc02d2d6884646cfc816f8153bf1c5375f2d2ba9e7034894b4e12509d\": container with ID starting with 7491868dc02d2d6884646cfc816f8153bf1c5375f2d2ba9e7034894b4e12509d not found: ID does not exist"
Sep 11 12:06:49.033469 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:49.033449    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"]
Sep 11 12:06:49.064649 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:49.064626    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-4a674b/noten-model-4a674b-kserve-69cc5dcbff-fhcg5"]
Sep 11 12:06:50.070837 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.070806    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e979668d-1a6c-41f2-90cd-5e3037482c3d" path="/var/lib/kubelet/pods/e979668d-1a6c-41f2-90cd-5e3037482c3d/volumes"
Sep 11 12:06:50.399563 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.399484    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"]
Sep 11 12:06:50.399939 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.399923    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e979668d-1a6c-41f2-90cd-5e3037482c3d" containerName="main"
Sep 11 12:06:50.400000 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.399940    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e979668d-1a6c-41f2-90cd-5e3037482c3d" containerName="main"
Sep 11 12:06:50.400000 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.399949    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9543c7f-5985-4d9a-a5c5-ad8a72415894" containerName="revoke-keys"
Sep 11 12:06:50.400000 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.399954    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9543c7f-5985-4d9a-a5c5-ad8a72415894" containerName="revoke-keys"
Sep 11 12:06:50.400166 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.400024    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e979668d-1a6c-41f2-90cd-5e3037482c3d" containerName="main"
Sep 11 12:06:50.400166 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.400036    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9543c7f-5985-4d9a-a5c5-ad8a72415894" containerName="revoke-keys"
Sep 11 12:06:50.404221 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.404203    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:50.407299 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.407268    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-24cfea9d-serving-cert\""
Sep 11 12:06:50.407670 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.407645    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 12:06:50.408276 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.408258    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rrxjh\""
Sep 11 12:06:50.417254 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.417230    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"]
Sep 11 12:06:50.424207 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.424185    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cbc380-h758j"]
Sep 11 12:06:50.432534 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.432502    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cbc380-h758j"]
Sep 11 12:06:50.481146 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.481119    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22eb4f18-edb9-4987-9ace-9551badcf1c7-openshift-service-ca\") pod \"maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:50.481273 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.481171    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-metrics-certs\") pod \"maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:50.481273 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.481208    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-maas-api-tls\") pod \"maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:50.481273 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.481255    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-722jl\" (UniqueName: \"kubernetes.io/projected/22eb4f18-edb9-4987-9ace-9551badcf1c7-kube-api-access-722jl\") pod \"maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:50.582117 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.582081    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22eb4f18-edb9-4987-9ace-9551badcf1c7-openshift-service-ca\") pod \"maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:50.582245 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.582129    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-metrics-certs\") pod \"maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:50.582245 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.582161    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-maas-api-tls\") pod \"maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:50.582245 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.582202    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-722jl\" (UniqueName: \"kubernetes.io/projected/22eb4f18-edb9-4987-9ace-9551badcf1c7-kube-api-access-722jl\") pod \"maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:50.582395 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:06:50.582308    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-serving-cert: secret "maas-api-e2e-mt-24cfea9d-serving-cert" not found
Sep 11 12:06:50.582395 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:06:50.582366    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-maas-api-tls podName:22eb4f18-edb9-4987-9ace-9551badcf1c7 nodeName:}" failed. No retries permitted until 2026-09-11 12:06:51.082351157 +0000 UTC m=+1133.504219523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-maas-api-tls") pod "maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd" (UID: "22eb4f18-edb9-4987-9ace-9551badcf1c7") : secret "maas-api-e2e-mt-24cfea9d-serving-cert" not found
Sep 11 12:06:50.582834 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.582815    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22eb4f18-edb9-4987-9ace-9551badcf1c7-openshift-service-ca\") pod \"maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:50.584727 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.584701    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-metrics-certs\") pod \"maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:50.620762 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:50.620738    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-722jl\" (UniqueName: \"kubernetes.io/projected/22eb4f18-edb9-4987-9ace-9551badcf1c7-kube-api-access-722jl\") pod \"maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:51.086429 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:51.086394    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-maas-api-tls\") pod \"maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:51.086780 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:06:51.086532    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-serving-cert: secret "maas-api-e2e-mt-24cfea9d-serving-cert" not found
Sep 11 12:06:51.086780 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:06:51.086607    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-maas-api-tls podName:22eb4f18-edb9-4987-9ace-9551badcf1c7 nodeName:}" failed. No retries permitted until 2026-09-11 12:06:52.086593163 +0000 UTC m=+1134.508461534 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-maas-api-tls") pod "maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd" (UID: "22eb4f18-edb9-4987-9ace-9551badcf1c7") : secret "maas-api-e2e-mt-24cfea9d-serving-cert" not found
Sep 11 12:06:52.069440 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:52.069408    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9543c7f-5985-4d9a-a5c5-ad8a72415894" path="/var/lib/kubelet/pods/e9543c7f-5985-4d9a-a5c5-ad8a72415894/volumes"
Sep 11 12:06:52.098721 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:52.098696    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-maas-api-tls\") pod \"maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:52.101121 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:52.101096    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-maas-api-tls\") pod \"maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:52.216574 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:52.216546    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:52.533166 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:06:52.533136    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22eb4f18_edb9_4987_9ace_9551badcf1c7.slice/crio-db7cfc0dd2372981be04a828605af8993a6c5063949890fd797550c0a1ee3737 WatchSource:0}: Error finding container db7cfc0dd2372981be04a828605af8993a6c5063949890fd797550c0a1ee3737: Status 404 returned error can't find the container with id db7cfc0dd2372981be04a828605af8993a6c5063949890fd797550c0a1ee3737
Sep 11 12:06:52.558185 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:52.558151    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"]
Sep 11 12:06:53.010134 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.010109    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd" event={"ID":"22eb4f18-edb9-4987-9ace-9551badcf1c7","Type":"ContainerStarted","Data":"44d40a0baf5b4d8fa3599a8dd9923b2d3c18d595d62d1d667366d519fa670c24"}
Sep 11 12:06:53.010256 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.010166    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd" event={"ID":"22eb4f18-edb9-4987-9ace-9551badcf1c7","Type":"ContainerStarted","Data":"db7cfc0dd2372981be04a828605af8993a6c5063949890fd797550c0a1ee3737"}
Sep 11 12:06:53.010325 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.010311    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:53.040958 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.040914    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd" podStartSLOduration=2.636186035 podStartE2EDuration="3.040900139s" totalImagesPullingTime="404.714104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:06:53.038651278 +0000 UTC m=+1135.460519673" watchObservedRunningTime="2026-09-11 12:06:53.040900139 +0000 UTC m=+1135.462768518"
Sep 11 12:06:53.222267 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.222179    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd" podUID="60eb5415-3c27-4862-b861-c660face056c" containerName="main" containerID="cri-o://42159ed3f603a3fa6511ace35479d323b394edae7f9e057a12964fccfdb9e652" gracePeriod=45
Sep 11 12:06:53.470806 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.470783    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:06:53.611255 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.611220    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-home\") pod \"60eb5415-3c27-4862-b861-c660face056c\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") "
Sep 11 12:06:53.611255 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.611255    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60eb5415-3c27-4862-b861-c660face056c-tls-certs\") pod \"60eb5415-3c27-4862-b861-c660face056c\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") "
Sep 11 12:06:53.611506 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.611325    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvwxd\" (UniqueName: \"kubernetes.io/projected/60eb5415-3c27-4862-b861-c660face056c-kube-api-access-bvwxd\") pod \"60eb5415-3c27-4862-b861-c660face056c\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") "
Sep 11 12:06:53.611506 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.611393    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-dshm\") pod \"60eb5415-3c27-4862-b861-c660face056c\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") "
Sep 11 12:06:53.611506 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.611412    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-model-cache\") pod \"60eb5415-3c27-4862-b861-c660face056c\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") "
Sep 11 12:06:53.611506 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.611451    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-tmp-dir\") pod \"60eb5415-3c27-4862-b861-c660face056c\" (UID: \"60eb5415-3c27-4862-b861-c660face056c\") "
Sep 11 12:06:53.611506 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.611488    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-home" (OuterVolumeSpecName: "home") pod "60eb5415-3c27-4862-b861-c660face056c" (UID: "60eb5415-3c27-4862-b861-c660face056c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:06:53.611804 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.611707    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-model-cache" (OuterVolumeSpecName: "model-cache") pod "60eb5415-3c27-4862-b861-c660face056c" (UID: "60eb5415-3c27-4862-b861-c660face056c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:06:53.611804 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.611790    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-home\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:53.611804 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.611788    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60eb5415-3c27-4862-b861-c660face056c" (UID: "60eb5415-3c27-4862-b861-c660face056c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:06:53.613841 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.613815    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60eb5415-3c27-4862-b861-c660face056c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60eb5415-3c27-4862-b861-c660face056c" (UID: "60eb5415-3c27-4862-b861-c660face056c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:06:53.613971 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.613889    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60eb5415-3c27-4862-b861-c660face056c-kube-api-access-bvwxd" (OuterVolumeSpecName: "kube-api-access-bvwxd") pod "60eb5415-3c27-4862-b861-c660face056c" (UID: "60eb5415-3c27-4862-b861-c660face056c"). InnerVolumeSpecName "kube-api-access-bvwxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:06:53.613971 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.613904    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-dshm" (OuterVolumeSpecName: "dshm") pod "60eb5415-3c27-4862-b861-c660face056c" (UID: "60eb5415-3c27-4862-b861-c660face056c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:06:53.712694 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.712672    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-dshm\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:53.712694 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.712691    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-model-cache\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:53.712845 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.712701    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60eb5415-3c27-4862-b861-c660face056c-tmp-dir\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:53.712845 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.712711    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60eb5415-3c27-4862-b861-c660face056c-tls-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:53.712845 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:53.712719    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvwxd\" (UniqueName: \"kubernetes.io/projected/60eb5415-3c27-4862-b861-c660face056c-kube-api-access-bvwxd\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:54.018466 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.018433    2569 generic.go:356] "Generic (PLEG): container finished" podID="60eb5415-3c27-4862-b861-c660face056c" containerID="42159ed3f603a3fa6511ace35479d323b394edae7f9e057a12964fccfdb9e652" exitCode=0
Sep 11 12:06:54.018657 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.018474    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"
Sep 11 12:06:54.018657 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.018516    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd" event={"ID":"60eb5415-3c27-4862-b861-c660face056c","Type":"ContainerDied","Data":"42159ed3f603a3fa6511ace35479d323b394edae7f9e057a12964fccfdb9e652"}
Sep 11 12:06:54.018657 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.018557    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd" event={"ID":"60eb5415-3c27-4862-b861-c660face056c","Type":"ContainerDied","Data":"23d8d7fec0dd5291592b1e3edab6960a3148f56887e48296846d295b0ff2703a"}
Sep 11 12:06:54.018657 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.018609    2569 scope.go:117] "RemoveContainer" containerID="42159ed3f603a3fa6511ace35479d323b394edae7f9e057a12964fccfdb9e652"
Sep 11 12:06:54.028772 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.028755    2569 scope.go:117] "RemoveContainer" containerID="42159ed3f603a3fa6511ace35479d323b394edae7f9e057a12964fccfdb9e652"
Sep 11 12:06:54.029052 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:06:54.029034    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42159ed3f603a3fa6511ace35479d323b394edae7f9e057a12964fccfdb9e652\": container with ID starting with 42159ed3f603a3fa6511ace35479d323b394edae7f9e057a12964fccfdb9e652 not found: ID does not exist" containerID="42159ed3f603a3fa6511ace35479d323b394edae7f9e057a12964fccfdb9e652"
Sep 11 12:06:54.029118 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.029063    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42159ed3f603a3fa6511ace35479d323b394edae7f9e057a12964fccfdb9e652"} err="failed to get container status \"42159ed3f603a3fa6511ace35479d323b394edae7f9e057a12964fccfdb9e652\": rpc error: code = NotFound desc = could not find container \"42159ed3f603a3fa6511ace35479d323b394edae7f9e057a12964fccfdb9e652\": container with ID starting with 42159ed3f603a3fa6511ace35479d323b394edae7f9e057a12964fccfdb9e652 not found: ID does not exist"
Sep 11 12:06:54.041369 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.041342    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"]
Sep 11 12:06:54.045574 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.045555    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-cbc380/test-model-cbc380-kserve-6f59ffbf5-qc8vd"]
Sep 11 12:06:54.069403 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.069378    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60eb5415-3c27-4862-b861-c660face056c" path="/var/lib/kubelet/pods/60eb5415-3c27-4862-b861-c660face056c/volumes"
Sep 11 12:06:54.245057 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.245025    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h" podUID="d108905c-bf96-46fa-b9a4-875bdc47cb74" containerName="main" containerID="cri-o://909270c1bed2c2caee9ebf9e053f0dbbd319ae93984845fed723cd4de86fec22" gracePeriod=45
Sep 11 12:06:54.494762 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.494738    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:06:54.621421 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.621338    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-dshm\") pod \"d108905c-bf96-46fa-b9a4-875bdc47cb74\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") "
Sep 11 12:06:54.621421 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.621410    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-tmp-dir\") pod \"d108905c-bf96-46fa-b9a4-875bdc47cb74\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") "
Sep 11 12:06:54.621660 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.621449    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cbpj\" (UniqueName: \"kubernetes.io/projected/d108905c-bf96-46fa-b9a4-875bdc47cb74-kube-api-access-7cbpj\") pod \"d108905c-bf96-46fa-b9a4-875bdc47cb74\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") "
Sep 11 12:06:54.621660 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.621479    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d108905c-bf96-46fa-b9a4-875bdc47cb74-tls-certs\") pod \"d108905c-bf96-46fa-b9a4-875bdc47cb74\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") "
Sep 11 12:06:54.621660 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.621548    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-model-cache\") pod \"d108905c-bf96-46fa-b9a4-875bdc47cb74\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") "
Sep 11 12:06:54.621660 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.621647    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-home\") pod \"d108905c-bf96-46fa-b9a4-875bdc47cb74\" (UID: \"d108905c-bf96-46fa-b9a4-875bdc47cb74\") "
Sep 11 12:06:54.621884 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.621735    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d108905c-bf96-46fa-b9a4-875bdc47cb74" (UID: "d108905c-bf96-46fa-b9a4-875bdc47cb74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:06:54.621972 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.621949    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-model-cache" (OuterVolumeSpecName: "model-cache") pod "d108905c-bf96-46fa-b9a4-875bdc47cb74" (UID: "d108905c-bf96-46fa-b9a4-875bdc47cb74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:06:54.622036 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.622009    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-home" (OuterVolumeSpecName: "home") pod "d108905c-bf96-46fa-b9a4-875bdc47cb74" (UID: "d108905c-bf96-46fa-b9a4-875bdc47cb74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:06:54.622111 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.622091    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-tmp-dir\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:54.622173 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.622115    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-model-cache\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:54.623740 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.623717    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d108905c-bf96-46fa-b9a4-875bdc47cb74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d108905c-bf96-46fa-b9a4-875bdc47cb74" (UID: "d108905c-bf96-46fa-b9a4-875bdc47cb74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:06:54.623850 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.623775    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-dshm" (OuterVolumeSpecName: "dshm") pod "d108905c-bf96-46fa-b9a4-875bdc47cb74" (UID: "d108905c-bf96-46fa-b9a4-875bdc47cb74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:06:54.623850 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.623787    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d108905c-bf96-46fa-b9a4-875bdc47cb74-kube-api-access-7cbpj" (OuterVolumeSpecName: "kube-api-access-7cbpj") pod "d108905c-bf96-46fa-b9a4-875bdc47cb74" (UID: "d108905c-bf96-46fa-b9a4-875bdc47cb74"). InnerVolumeSpecName "kube-api-access-7cbpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:06:54.723001 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.722971    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-dshm\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:54.723001 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.722998    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cbpj\" (UniqueName: \"kubernetes.io/projected/d108905c-bf96-46fa-b9a4-875bdc47cb74-kube-api-access-7cbpj\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:54.723208 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.723012    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d108905c-bf96-46fa-b9a4-875bdc47cb74-tls-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:54.723208 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:54.723026    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d108905c-bf96-46fa-b9a4-875bdc47cb74-home\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:06:55.027830 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:55.027802    2569 generic.go:356] "Generic (PLEG): container finished" podID="d108905c-bf96-46fa-b9a4-875bdc47cb74" containerID="909270c1bed2c2caee9ebf9e053f0dbbd319ae93984845fed723cd4de86fec22" exitCode=0
Sep 11 12:06:55.027999 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:55.027847    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"
Sep 11 12:06:55.027999 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:55.027886    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h" event={"ID":"d108905c-bf96-46fa-b9a4-875bdc47cb74","Type":"ContainerDied","Data":"909270c1bed2c2caee9ebf9e053f0dbbd319ae93984845fed723cd4de86fec22"}
Sep 11 12:06:55.027999 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:55.027930    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h" event={"ID":"d108905c-bf96-46fa-b9a4-875bdc47cb74","Type":"ContainerDied","Data":"95b5b56b5927f4df6fcb14d2bd9661c8fd187d665aedbd80315c034df9cb4687"}
Sep 11 12:06:55.027999 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:55.027952    2569 scope.go:117] "RemoveContainer" containerID="909270c1bed2c2caee9ebf9e053f0dbbd319ae93984845fed723cd4de86fec22"
Sep 11 12:06:55.038640 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:55.038616    2569 scope.go:117] "RemoveContainer" containerID="909270c1bed2c2caee9ebf9e053f0dbbd319ae93984845fed723cd4de86fec22"
Sep 11 12:06:55.038904 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:06:55.038881    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909270c1bed2c2caee9ebf9e053f0dbbd319ae93984845fed723cd4de86fec22\": container with ID starting with 909270c1bed2c2caee9ebf9e053f0dbbd319ae93984845fed723cd4de86fec22 not found: ID does not exist" containerID="909270c1bed2c2caee9ebf9e053f0dbbd319ae93984845fed723cd4de86fec22"
Sep 11 12:06:55.039002 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:55.038908    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909270c1bed2c2caee9ebf9e053f0dbbd319ae93984845fed723cd4de86fec22"} err="failed to get container status \"909270c1bed2c2caee9ebf9e053f0dbbd319ae93984845fed723cd4de86fec22\": rpc error: code = NotFound desc = could not find container \"909270c1bed2c2caee9ebf9e053f0dbbd319ae93984845fed723cd4de86fec22\": container with ID starting with 909270c1bed2c2caee9ebf9e053f0dbbd319ae93984845fed723cd4de86fec22 not found: ID does not exist"
Sep 11 12:06:55.055718 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:55.055678    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"]
Sep 11 12:06:55.060210 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:55.060186    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-12d2a9/test-model-12d2a9-kserve-5d546b98d7-dkd6h"]
Sep 11 12:06:56.069687 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:56.069656    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d108905c-bf96-46fa-b9a4-875bdc47cb74" path="/var/lib/kubelet/pods/d108905c-bf96-46fa-b9a4-875bdc47cb74/volumes"
Sep 11 12:06:57.611997 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:57.611963    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6"]
Sep 11 12:06:57.612388 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:57.612373    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60eb5415-3c27-4862-b861-c660face056c" containerName="main"
Sep 11 12:06:57.612429 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:57.612389    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="60eb5415-3c27-4862-b861-c660face056c" containerName="main"
Sep 11 12:06:57.612465 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:57.612430    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d108905c-bf96-46fa-b9a4-875bdc47cb74" containerName="main"
Sep 11 12:06:57.612465 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:57.612436    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d108905c-bf96-46fa-b9a4-875bdc47cb74" containerName="main"
Sep 11 12:06:57.612530 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:57.612496    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d108905c-bf96-46fa-b9a4-875bdc47cb74" containerName="main"
Sep 11 12:06:57.612530 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:57.612506    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="60eb5415-3c27-4862-b861-c660face056c" containerName="main"
Sep 11 12:06:57.615505 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:57.615486    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6"
Sep 11 12:06:57.618311 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:57.618290    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xkpf\""
Sep 11 12:06:57.627895 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:57.627874    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6"]
Sep 11 12:06:57.753964 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:57.753928    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e70067-4cfb-45ce-b41d-00e6a84ff234-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6\" (UID: \"47e70067-4cfb-45ce-b41d-00e6a84ff234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6"
Sep 11 12:06:57.854730 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:57.854693    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e70067-4cfb-45ce-b41d-00e6a84ff234-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6\" (UID: \"47e70067-4cfb-45ce-b41d-00e6a84ff234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6"
Sep 11 12:06:57.855345 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:57.855324    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e70067-4cfb-45ce-b41d-00e6a84ff234-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6\" (UID: \"47e70067-4cfb-45ce-b41d-00e6a84ff234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6"
Sep 11 12:06:57.927312 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:57.927240    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6"
Sep 11 12:06:58.200296 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:58.200267    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6"]
Sep 11 12:06:58.201089 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:06:58.201067    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e70067_4cfb_45ce_b41d_00e6a84ff234.slice/crio-f0f37bd665be726d2ae65a7a54584f073fa192337d386892d3fa15da334b6d22 WatchSource:0}: Error finding container f0f37bd665be726d2ae65a7a54584f073fa192337d386892d3fa15da334b6d22: Status 404 returned error can't find the container with id f0f37bd665be726d2ae65a7a54584f073fa192337d386892d3fa15da334b6d22
Sep 11 12:06:59.024174 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:59.024103    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:06:59.053451 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:59.053421    2569 generic.go:356] "Generic (PLEG): container finished" podID="47e70067-4cfb-45ce-b41d-00e6a84ff234" containerID="20fe6dc28223edb8000b918c94aef0a35885e547bc40b28b42ae6a7f03ee705a" exitCode=0
Sep 11 12:06:59.053609 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:59.053505    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6" event={"ID":"47e70067-4cfb-45ce-b41d-00e6a84ff234","Type":"ContainerDied","Data":"20fe6dc28223edb8000b918c94aef0a35885e547bc40b28b42ae6a7f03ee705a"}
Sep 11 12:06:59.053609 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:06:59.053556    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6" event={"ID":"47e70067-4cfb-45ce-b41d-00e6a84ff234","Type":"ContainerStarted","Data":"f0f37bd665be726d2ae65a7a54584f073fa192337d386892d3fa15da334b6d22"}
Sep 11 12:07:00.194712 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:00.194689    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6"
Sep 11 12:07:00.276894 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:00.276863    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e70067-4cfb-45ce-b41d-00e6a84ff234-maas-api-service-ca\") pod \"47e70067-4cfb-45ce-b41d-00e6a84ff234\" (UID: \"47e70067-4cfb-45ce-b41d-00e6a84ff234\") "
Sep 11 12:07:00.277204 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:00.277183    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e70067-4cfb-45ce-b41d-00e6a84ff234-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "47e70067-4cfb-45ce-b41d-00e6a84ff234" (UID: "47e70067-4cfb-45ce-b41d-00e6a84ff234"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:07:00.377952 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:00.377874    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e70067-4cfb-45ce-b41d-00e6a84ff234-maas-api-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:07:01.066169 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:01.066136    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6"
Sep 11 12:07:01.066337 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:01.066183    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6" event={"ID":"47e70067-4cfb-45ce-b41d-00e6a84ff234","Type":"ContainerDied","Data":"f0f37bd665be726d2ae65a7a54584f073fa192337d386892d3fa15da334b6d22"}
Sep 11 12:07:01.066337 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:01.066226    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0f37bd665be726d2ae65a7a54584f073fa192337d386892d3fa15da334b6d22"
Sep 11 12:07:08.284181 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:08.284113    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"]
Sep 11 12:07:08.285086 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:08.284499    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq" podUID="dadfdeb4-ac33-44f0-bec2-9ff59086bb21" containerName="payload-pre-processing" containerID="cri-o://67c9f13371cffe2615ee73f1159397b8dfdf8513ffb2223706b273029396d416" gracePeriod=30
Sep 11 12:07:08.489849 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:08.489818    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6"]
Sep 11 12:07:08.493219 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:08.493197    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12d2a9-4wcd6"]
Sep 11 12:07:08.546081 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:08.546022    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"
Sep 11 12:07:08.652486 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:08.652458    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dadfdeb4-ac33-44f0-bec2-9ff59086bb21-plugins-config-volume\") pod \"dadfdeb4-ac33-44f0-bec2-9ff59086bb21\" (UID: \"dadfdeb4-ac33-44f0-bec2-9ff59086bb21\") "
Sep 11 12:07:08.652636 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:08.652526    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv5zk\" (UniqueName: \"kubernetes.io/projected/dadfdeb4-ac33-44f0-bec2-9ff59086bb21-kube-api-access-nv5zk\") pod \"dadfdeb4-ac33-44f0-bec2-9ff59086bb21\" (UID: \"dadfdeb4-ac33-44f0-bec2-9ff59086bb21\") "
Sep 11 12:07:08.652822 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:08.652802    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dadfdeb4-ac33-44f0-bec2-9ff59086bb21-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dadfdeb4-ac33-44f0-bec2-9ff59086bb21" (UID: "dadfdeb4-ac33-44f0-bec2-9ff59086bb21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:07:08.654505 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:08.654487    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dadfdeb4-ac33-44f0-bec2-9ff59086bb21-kube-api-access-nv5zk" (OuterVolumeSpecName: "kube-api-access-nv5zk") pod "dadfdeb4-ac33-44f0-bec2-9ff59086bb21" (UID: "dadfdeb4-ac33-44f0-bec2-9ff59086bb21"). InnerVolumeSpecName "kube-api-access-nv5zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:07:08.753324 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:08.753299    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dadfdeb4-ac33-44f0-bec2-9ff59086bb21-plugins-config-volume\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:07:08.753324 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:08.753322    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv5zk\" (UniqueName: \"kubernetes.io/projected/dadfdeb4-ac33-44f0-bec2-9ff59086bb21-kube-api-access-nv5zk\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:07:09.112492 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:09.112454    2569 generic.go:356] "Generic (PLEG): container finished" podID="dadfdeb4-ac33-44f0-bec2-9ff59086bb21" containerID="67c9f13371cffe2615ee73f1159397b8dfdf8513ffb2223706b273029396d416" exitCode=0
Sep 11 12:07:09.112674 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:09.112511    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq" event={"ID":"dadfdeb4-ac33-44f0-bec2-9ff59086bb21","Type":"ContainerDied","Data":"67c9f13371cffe2615ee73f1159397b8dfdf8513ffb2223706b273029396d416"}
Sep 11 12:07:09.112674 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:09.112522    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"
Sep 11 12:07:09.112674 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:09.112551    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq" event={"ID":"dadfdeb4-ac33-44f0-bec2-9ff59086bb21","Type":"ContainerDied","Data":"cf07bab06f0c8409f59c8a69a38e54cfd04379ac87c1a403a43fd359368b9b60"}
Sep 11 12:07:09.112674 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:09.112572    2569 scope.go:117] "RemoveContainer" containerID="67c9f13371cffe2615ee73f1159397b8dfdf8513ffb2223706b273029396d416"
Sep 11 12:07:09.122343 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:09.122325    2569 scope.go:117] "RemoveContainer" containerID="67c9f13371cffe2615ee73f1159397b8dfdf8513ffb2223706b273029396d416"
Sep 11 12:07:09.122627 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:07:09.122607    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67c9f13371cffe2615ee73f1159397b8dfdf8513ffb2223706b273029396d416\": container with ID starting with 67c9f13371cffe2615ee73f1159397b8dfdf8513ffb2223706b273029396d416 not found: ID does not exist" containerID="67c9f13371cffe2615ee73f1159397b8dfdf8513ffb2223706b273029396d416"
Sep 11 12:07:09.122687 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:09.122635    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67c9f13371cffe2615ee73f1159397b8dfdf8513ffb2223706b273029396d416"} err="failed to get container status \"67c9f13371cffe2615ee73f1159397b8dfdf8513ffb2223706b273029396d416\": rpc error: code = NotFound desc = could not find container \"67c9f13371cffe2615ee73f1159397b8dfdf8513ffb2223706b273029396d416\": container with ID starting with 67c9f13371cffe2615ee73f1159397b8dfdf8513ffb2223706b273029396d416 not found: ID does not exist"
Sep 11 12:07:09.136724 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:09.136700    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"]
Sep 11 12:07:09.143460 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:09.143438    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12d2a9-595d58c794-xs4gq"]
Sep 11 12:07:10.070224 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:10.070191    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47e70067-4cfb-45ce-b41d-00e6a84ff234" path="/var/lib/kubelet/pods/47e70067-4cfb-45ce-b41d-00e6a84ff234/volumes"
Sep 11 12:07:10.070756 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:10.070737    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dadfdeb4-ac33-44f0-bec2-9ff59086bb21" path="/var/lib/kubelet/pods/dadfdeb4-ac33-44f0-bec2-9ff59086bb21/volumes"
Sep 11 12:07:21.866567 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:21.866481    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"]
Sep 11 12:07:21.867015 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:21.866885    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47e70067-4cfb-45ce-b41d-00e6a84ff234" containerName="revoke-keys"
Sep 11 12:07:21.867015 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:21.866897    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="47e70067-4cfb-45ce-b41d-00e6a84ff234" containerName="revoke-keys"
Sep 11 12:07:21.867015 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:21.866914    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dadfdeb4-ac33-44f0-bec2-9ff59086bb21" containerName="payload-pre-processing"
Sep 11 12:07:21.867015 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:21.866919    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dadfdeb4-ac33-44f0-bec2-9ff59086bb21" containerName="payload-pre-processing"
Sep 11 12:07:21.867015 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:21.866977    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dadfdeb4-ac33-44f0-bec2-9ff59086bb21" containerName="payload-pre-processing"
Sep 11 12:07:21.867015 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:21.866987    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="47e70067-4cfb-45ce-b41d-00e6a84ff234" containerName="revoke-keys"
Sep 11 12:07:21.870291 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:21.870266    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:07:21.874549 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:21.874527    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-af081c-serving-cert\""
Sep 11 12:07:21.887113 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:21.887089    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"]
Sep 11 12:07:21.966847 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:21.966821    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6093502b-ce5d-42aa-9c45-41fe039002c3-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h\" (UID: \"6093502b-ce5d-42aa-9c45-41fe039002c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:07:21.966965 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:21.966852    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6093502b-ce5d-42aa-9c45-41fe039002c3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h\" (UID: \"6093502b-ce5d-42aa-9c45-41fe039002c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:07:21.966965 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:21.966908    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fkt4\" (UniqueName: \"kubernetes.io/projected/6093502b-ce5d-42aa-9c45-41fe039002c3-kube-api-access-5fkt4\") pod \"maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h\" (UID: \"6093502b-ce5d-42aa-9c45-41fe039002c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:07:21.966965 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:21.966958    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6093502b-ce5d-42aa-9c45-41fe039002c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h\" (UID: \"6093502b-ce5d-42aa-9c45-41fe039002c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:07:22.068186 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:22.068158    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6093502b-ce5d-42aa-9c45-41fe039002c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h\" (UID: \"6093502b-ce5d-42aa-9c45-41fe039002c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:07:22.068321 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:22.068215    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6093502b-ce5d-42aa-9c45-41fe039002c3-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h\" (UID: \"6093502b-ce5d-42aa-9c45-41fe039002c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:07:22.068381 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:07:22.068321    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-serving-cert: secret "maas-api-e2e-shared-a-w3-af081c-serving-cert" not found
Sep 11 12:07:22.068381 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:22.068336    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6093502b-ce5d-42aa-9c45-41fe039002c3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h\" (UID: \"6093502b-ce5d-42aa-9c45-41fe039002c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:07:22.068497 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:07:22.068389    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6093502b-ce5d-42aa-9c45-41fe039002c3-maas-api-tls podName:6093502b-ce5d-42aa-9c45-41fe039002c3 nodeName:}" failed. No retries permitted until 2026-09-11 12:07:22.568367172 +0000 UTC m=+1164.990235536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6093502b-ce5d-42aa-9c45-41fe039002c3-maas-api-tls") pod "maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h" (UID: "6093502b-ce5d-42aa-9c45-41fe039002c3") : secret "maas-api-e2e-shared-a-w3-af081c-serving-cert" not found
Sep 11 12:07:22.068497 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:22.068462    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fkt4\" (UniqueName: \"kubernetes.io/projected/6093502b-ce5d-42aa-9c45-41fe039002c3-kube-api-access-5fkt4\") pod \"maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h\" (UID: \"6093502b-ce5d-42aa-9c45-41fe039002c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:07:22.069044 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:22.069023    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6093502b-ce5d-42aa-9c45-41fe039002c3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h\" (UID: \"6093502b-ce5d-42aa-9c45-41fe039002c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:07:22.071138 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:22.071118    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6093502b-ce5d-42aa-9c45-41fe039002c3-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h\" (UID: \"6093502b-ce5d-42aa-9c45-41fe039002c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:07:22.079875 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:22.079857    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fkt4\" (UniqueName: \"kubernetes.io/projected/6093502b-ce5d-42aa-9c45-41fe039002c3-kube-api-access-5fkt4\") pod \"maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h\" (UID: \"6093502b-ce5d-42aa-9c45-41fe039002c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:07:22.571919 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:22.571883    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6093502b-ce5d-42aa-9c45-41fe039002c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h\" (UID: \"6093502b-ce5d-42aa-9c45-41fe039002c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:07:22.574212 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:22.574187    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6093502b-ce5d-42aa-9c45-41fe039002c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h\" (UID: \"6093502b-ce5d-42aa-9c45-41fe039002c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:07:22.781172 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:22.781139    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:07:23.061702 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:23.061673    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"]
Sep 11 12:07:23.063634 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:07:23.063609    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6093502b_ce5d_42aa_9c45_41fe039002c3.slice/crio-0bc6c8b77fd4eef1d4a86d4a31abe18ddb5cbba669dc97c3aafef8db00e2d792 WatchSource:0}: Error finding container 0bc6c8b77fd4eef1d4a86d4a31abe18ddb5cbba669dc97c3aafef8db00e2d792: Status 404 returned error can't find the container with id 0bc6c8b77fd4eef1d4a86d4a31abe18ddb5cbba669dc97c3aafef8db00e2d792
Sep 11 12:07:23.198370 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:23.198293    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h" event={"ID":"6093502b-ce5d-42aa-9c45-41fe039002c3","Type":"ContainerStarted","Data":"0bc6c8b77fd4eef1d4a86d4a31abe18ddb5cbba669dc97c3aafef8db00e2d792"}
Sep 11 12:07:24.206108 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:24.206075    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h" event={"ID":"6093502b-ce5d-42aa-9c45-41fe039002c3","Type":"ContainerStarted","Data":"3554bb81f9a1969796c40602a45a21f6c38890de981ae92fb5eb50002d787171"}
Sep 11 12:07:24.206527 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:24.206139    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:07:24.225778 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:24.225727    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h" podStartSLOduration=2.672741154 podStartE2EDuration="3.225711876s" totalImagesPullingTime="552.970722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:07:24.223858146 +0000 UTC m=+1166.645726550" watchObservedRunningTime="2026-09-11 12:07:24.225711876 +0000 UTC m=+1166.647580256"
Sep 11 12:07:30.218888 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:30.218860    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:07:34.208696 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:34.208663    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"]
Sep 11 12:07:34.209113 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:34.208910    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd" podUID="22eb4f18-edb9-4987-9ace-9551badcf1c7" containerName="maas-api" containerID="cri-o://44d40a0baf5b4d8fa3599a8dd9923b2d3c18d595d62d1d667366d519fa670c24" gracePeriod=30
Sep 11 12:07:34.459175 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:34.459118    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:07:34.569705 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:34.569677    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-metrics-certs\") pod \"22eb4f18-edb9-4987-9ace-9551badcf1c7\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") "
Sep 11 12:07:34.569856 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:34.569721    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-722jl\" (UniqueName: \"kubernetes.io/projected/22eb4f18-edb9-4987-9ace-9551badcf1c7-kube-api-access-722jl\") pod \"22eb4f18-edb9-4987-9ace-9551badcf1c7\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") "
Sep 11 12:07:34.569856 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:34.569755    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-maas-api-tls\") pod \"22eb4f18-edb9-4987-9ace-9551badcf1c7\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") "
Sep 11 12:07:34.569856 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:34.569847    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22eb4f18-edb9-4987-9ace-9551badcf1c7-openshift-service-ca\") pod \"22eb4f18-edb9-4987-9ace-9551badcf1c7\" (UID: \"22eb4f18-edb9-4987-9ace-9551badcf1c7\") "
Sep 11 12:07:34.570302 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:34.570271    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22eb4f18-edb9-4987-9ace-9551badcf1c7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "22eb4f18-edb9-4987-9ace-9551badcf1c7" (UID: "22eb4f18-edb9-4987-9ace-9551badcf1c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:07:34.571939 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:34.571911    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22eb4f18-edb9-4987-9ace-9551badcf1c7-kube-api-access-722jl" (OuterVolumeSpecName: "kube-api-access-722jl") pod "22eb4f18-edb9-4987-9ace-9551badcf1c7" (UID: "22eb4f18-edb9-4987-9ace-9551badcf1c7"). InnerVolumeSpecName "kube-api-access-722jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:07:34.572047 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:34.571978    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "22eb4f18-edb9-4987-9ace-9551badcf1c7" (UID: "22eb4f18-edb9-4987-9ace-9551badcf1c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:07:34.572047 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:34.571993    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "22eb4f18-edb9-4987-9ace-9551badcf1c7" (UID: "22eb4f18-edb9-4987-9ace-9551badcf1c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:07:34.670532 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:34.670502    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22eb4f18-edb9-4987-9ace-9551badcf1c7-openshift-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:07:34.670532 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:34.670529    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-metrics-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:07:34.670532 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:34.670538    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-722jl\" (UniqueName: \"kubernetes.io/projected/22eb4f18-edb9-4987-9ace-9551badcf1c7-kube-api-access-722jl\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:07:34.670743 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:34.670547    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22eb4f18-edb9-4987-9ace-9551badcf1c7-maas-api-tls\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:07:35.272184 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:35.272151    2569 generic.go:356] "Generic (PLEG): container finished" podID="22eb4f18-edb9-4987-9ace-9551badcf1c7" containerID="44d40a0baf5b4d8fa3599a8dd9923b2d3c18d595d62d1d667366d519fa670c24" exitCode=0
Sep 11 12:07:35.272657 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:35.272195    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"
Sep 11 12:07:35.272657 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:35.272234    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd" event={"ID":"22eb4f18-edb9-4987-9ace-9551badcf1c7","Type":"ContainerDied","Data":"44d40a0baf5b4d8fa3599a8dd9923b2d3c18d595d62d1d667366d519fa670c24"}
Sep 11 12:07:35.272657 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:35.272295    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd" event={"ID":"22eb4f18-edb9-4987-9ace-9551badcf1c7","Type":"ContainerDied","Data":"db7cfc0dd2372981be04a828605af8993a6c5063949890fd797550c0a1ee3737"}
Sep 11 12:07:35.272657 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:35.272329    2569 scope.go:117] "RemoveContainer" containerID="44d40a0baf5b4d8fa3599a8dd9923b2d3c18d595d62d1d667366d519fa670c24"
Sep 11 12:07:35.281831 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:35.281815    2569 scope.go:117] "RemoveContainer" containerID="44d40a0baf5b4d8fa3599a8dd9923b2d3c18d595d62d1d667366d519fa670c24"
Sep 11 12:07:35.282099 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:07:35.282081    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d40a0baf5b4d8fa3599a8dd9923b2d3c18d595d62d1d667366d519fa670c24\": container with ID starting with 44d40a0baf5b4d8fa3599a8dd9923b2d3c18d595d62d1d667366d519fa670c24 not found: ID does not exist" containerID="44d40a0baf5b4d8fa3599a8dd9923b2d3c18d595d62d1d667366d519fa670c24"
Sep 11 12:07:35.282172 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:35.282105    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d40a0baf5b4d8fa3599a8dd9923b2d3c18d595d62d1d667366d519fa670c24"} err="failed to get container status \"44d40a0baf5b4d8fa3599a8dd9923b2d3c18d595d62d1d667366d519fa670c24\": rpc error: code = NotFound desc = could not find container \"44d40a0baf5b4d8fa3599a8dd9923b2d3c18d595d62d1d667366d519fa670c24\": container with ID starting with 44d40a0baf5b4d8fa3599a8dd9923b2d3c18d595d62d1d667366d519fa670c24 not found: ID does not exist"
Sep 11 12:07:35.336141 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:35.336115    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"]
Sep 11 12:07:35.353536 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:35.353507    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-24cfea9d-5cc4cdf49b-q4zzd"]
Sep 11 12:07:36.069551 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:36.069518    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22eb4f18-edb9-4987-9ace-9551badcf1c7" path="/var/lib/kubelet/pods/22eb4f18-edb9-4987-9ace-9551badcf1c7/volumes"
Sep 11 12:07:50.588733 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.588693    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"]
Sep 11 12:07:50.591101 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.589109    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22eb4f18-edb9-4987-9ace-9551badcf1c7" containerName="maas-api"
Sep 11 12:07:50.591101 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.589121    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="22eb4f18-edb9-4987-9ace-9551badcf1c7" containerName="maas-api"
Sep 11 12:07:50.591101 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.589191    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="22eb4f18-edb9-4987-9ace-9551badcf1c7" containerName="maas-api"
Sep 11 12:07:50.591978 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.591963    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:07:50.595133 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.595110    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-d844cc-serving-cert\""
Sep 11 12:07:50.604223 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.604203    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"]
Sep 11 12:07:50.712666 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.712639    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cdf6e64-d037-4572-bfe5-72a763f0766d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj\" (UID: \"3cdf6e64-d037-4572-bfe5-72a763f0766d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:07:50.712781 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.712708    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cdf6e64-d037-4572-bfe5-72a763f0766d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj\" (UID: \"3cdf6e64-d037-4572-bfe5-72a763f0766d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:07:50.712781 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.712745    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lmtg\" (UniqueName: \"kubernetes.io/projected/3cdf6e64-d037-4572-bfe5-72a763f0766d-kube-api-access-7lmtg\") pod \"maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj\" (UID: \"3cdf6e64-d037-4572-bfe5-72a763f0766d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:07:50.712893 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.712797    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cdf6e64-d037-4572-bfe5-72a763f0766d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj\" (UID: \"3cdf6e64-d037-4572-bfe5-72a763f0766d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:07:50.813280 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.813253    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cdf6e64-d037-4572-bfe5-72a763f0766d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj\" (UID: \"3cdf6e64-d037-4572-bfe5-72a763f0766d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:07:50.813398 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.813305    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cdf6e64-d037-4572-bfe5-72a763f0766d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj\" (UID: \"3cdf6e64-d037-4572-bfe5-72a763f0766d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:07:50.813398 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.813332    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lmtg\" (UniqueName: \"kubernetes.io/projected/3cdf6e64-d037-4572-bfe5-72a763f0766d-kube-api-access-7lmtg\") pod \"maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj\" (UID: \"3cdf6e64-d037-4572-bfe5-72a763f0766d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:07:50.813398 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.813365    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cdf6e64-d037-4572-bfe5-72a763f0766d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj\" (UID: \"3cdf6e64-d037-4572-bfe5-72a763f0766d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:07:50.814098 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.814073    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cdf6e64-d037-4572-bfe5-72a763f0766d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj\" (UID: \"3cdf6e64-d037-4572-bfe5-72a763f0766d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:07:50.815873 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.815852    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cdf6e64-d037-4572-bfe5-72a763f0766d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj\" (UID: \"3cdf6e64-d037-4572-bfe5-72a763f0766d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:07:50.815960 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.815923    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cdf6e64-d037-4572-bfe5-72a763f0766d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj\" (UID: \"3cdf6e64-d037-4572-bfe5-72a763f0766d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:07:50.830772 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.830748    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lmtg\" (UniqueName: \"kubernetes.io/projected/3cdf6e64-d037-4572-bfe5-72a763f0766d-kube-api-access-7lmtg\") pod \"maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj\" (UID: \"3cdf6e64-d037-4572-bfe5-72a763f0766d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:07:50.903460 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:50.903398    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:07:51.214887 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:07:51.214860    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cdf6e64_d037_4572_bfe5_72a763f0766d.slice/crio-cd1d6cda183f46d44a1d49b6b86ac37b09b052347c8dc33259f571dc84afe1ca WatchSource:0}: Error finding container cd1d6cda183f46d44a1d49b6b86ac37b09b052347c8dc33259f571dc84afe1ca: Status 404 returned error can't find the container with id cd1d6cda183f46d44a1d49b6b86ac37b09b052347c8dc33259f571dc84afe1ca
Sep 11 12:07:51.218353 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:51.218322    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"]
Sep 11 12:07:51.371923 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:51.371891    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj" event={"ID":"3cdf6e64-d037-4572-bfe5-72a763f0766d","Type":"ContainerStarted","Data":"cd1d6cda183f46d44a1d49b6b86ac37b09b052347c8dc33259f571dc84afe1ca"}
Sep 11 12:07:52.379169 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:52.379135    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj" event={"ID":"3cdf6e64-d037-4572-bfe5-72a763f0766d","Type":"ContainerStarted","Data":"6fcdd9ab6b5879df1d58a4516081dcf1ee145ed71b8890a873b26bb08a316da6"}
Sep 11 12:07:52.379683 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:52.379198    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:07:52.409793 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:52.409735    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj" podStartSLOduration=1.8825463199999999 podStartE2EDuration="2.409714932s" totalImagesPullingTime="527.168612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:07:52.409469452 +0000 UTC m=+1194.831337857" watchObservedRunningTime="2026-09-11 12:07:52.409714932 +0000 UTC m=+1194.831583313"
Sep 11 12:07:58.141649 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:58.141619    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 12:07:58.143573 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:58.143551    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 12:07:58.391123 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:07:58.391098    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:08:01.670739 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:01.670704    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"]
Sep 11 12:08:01.674948 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:01.674926    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:08:01.677477 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:01.677456    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-45352b-serving-cert\""
Sep 11 12:08:01.687660 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:01.687638    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"]
Sep 11 12:08:01.813075 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:01.813049    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh\" (UID: \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:08:01.813220 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:01.813085    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh\" (UID: \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:08:01.813220 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:01.813110    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bshz\" (UniqueName: \"kubernetes.io/projected/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-kube-api-access-2bshz\") pod \"maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh\" (UID: \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:08:01.813302 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:01.813216    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh\" (UID: \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:08:01.914309 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:01.914275    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh\" (UID: \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:08:01.914537 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:01.914320    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh\" (UID: \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:08:01.914537 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:01.914342    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bshz\" (UniqueName: \"kubernetes.io/projected/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-kube-api-access-2bshz\") pod \"maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh\" (UID: \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:08:01.914537 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:01.914393    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh\" (UID: \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:08:01.914778 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:08:01.914564    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-serving-cert: secret "maas-api-e2e-shared-a-w2-45352b-serving-cert" not found
Sep 11 12:08:01.914778 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:08:01.914662    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-maas-api-tls podName:7fe9c22f-b313-419e-9eb0-dcac0553ab2c nodeName:}" failed. No retries permitted until 2026-09-11 12:08:02.414642314 +0000 UTC m=+1204.836510689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-maas-api-tls") pod "maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh" (UID: "7fe9c22f-b313-419e-9eb0-dcac0553ab2c") : secret "maas-api-e2e-shared-a-w2-45352b-serving-cert" not found
Sep 11 12:08:01.915066 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:01.915044    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh\" (UID: \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:08:01.916858 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:01.916837    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh\" (UID: \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:08:01.925283 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:01.925227    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bshz\" (UniqueName: \"kubernetes.io/projected/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-kube-api-access-2bshz\") pod \"maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh\" (UID: \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:08:02.419872 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:02.419839    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh\" (UID: \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:08:02.422400 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:02.422369    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh\" (UID: \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:08:02.587836 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:02.587802    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:08:02.861432 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:02.861397    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"]
Sep 11 12:08:02.862800 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:08:02.862769    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe9c22f_b313_419e_9eb0_dcac0553ab2c.slice/crio-437d89b09904442e1bccb6a9e0f1d215919586f1f4ea3b3590a512e49b3abd47 WatchSource:0}: Error finding container 437d89b09904442e1bccb6a9e0f1d215919586f1f4ea3b3590a512e49b3abd47: Status 404 returned error can't find the container with id 437d89b09904442e1bccb6a9e0f1d215919586f1f4ea3b3590a512e49b3abd47
Sep 11 12:08:03.448102 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:03.448068    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh" event={"ID":"7fe9c22f-b313-419e-9eb0-dcac0553ab2c","Type":"ContainerStarted","Data":"437d89b09904442e1bccb6a9e0f1d215919586f1f4ea3b3590a512e49b3abd47"}
Sep 11 12:08:04.455320 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:04.455278    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh" event={"ID":"7fe9c22f-b313-419e-9eb0-dcac0553ab2c","Type":"ContainerStarted","Data":"cc65d5d415e22a5300c899bfc9f56d7f94c764252c2841b15234bffe7cbb994f"}
Sep 11 12:08:04.455680 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:04.455358    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:08:04.479110 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:04.479058    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh" podStartSLOduration=2.416122552 podStartE2EDuration="3.479042144s" totalImagesPullingTime="1.062919592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:08:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:08:04.47561275 +0000 UTC m=+1206.897481147" watchObservedRunningTime="2026-09-11 12:08:04.479042144 +0000 UTC m=+1206.900910568"
Sep 11 12:08:10.466916 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:10.466885    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:08:30.141343 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.141309    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"]
Sep 11 12:08:30.145548 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.145526    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:08:30.148071 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.148047    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-f7851b-serving-cert\""
Sep 11 12:08:30.159689 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.159665    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"]
Sep 11 12:08:30.263634 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.263577    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05472180-97a7-40f5-9801-c39de1b24470-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq\" (UID: \"05472180-97a7-40f5-9801-c39de1b24470\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:08:30.263822 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.263703    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05472180-97a7-40f5-9801-c39de1b24470-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq\" (UID: \"05472180-97a7-40f5-9801-c39de1b24470\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:08:30.263822 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.263730    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05472180-97a7-40f5-9801-c39de1b24470-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq\" (UID: \"05472180-97a7-40f5-9801-c39de1b24470\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:08:30.263822 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.263801    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqndd\" (UniqueName: \"kubernetes.io/projected/05472180-97a7-40f5-9801-c39de1b24470-kube-api-access-nqndd\") pod \"maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq\" (UID: \"05472180-97a7-40f5-9801-c39de1b24470\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:08:30.364484 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.364447    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05472180-97a7-40f5-9801-c39de1b24470-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq\" (UID: \"05472180-97a7-40f5-9801-c39de1b24470\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:08:30.364689 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.364490    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05472180-97a7-40f5-9801-c39de1b24470-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq\" (UID: \"05472180-97a7-40f5-9801-c39de1b24470\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:08:30.364689 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.364517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqndd\" (UniqueName: \"kubernetes.io/projected/05472180-97a7-40f5-9801-c39de1b24470-kube-api-access-nqndd\") pod \"maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq\" (UID: \"05472180-97a7-40f5-9801-c39de1b24470\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:08:30.364689 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.364555    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05472180-97a7-40f5-9801-c39de1b24470-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq\" (UID: \"05472180-97a7-40f5-9801-c39de1b24470\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:08:30.365133 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.365099    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05472180-97a7-40f5-9801-c39de1b24470-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq\" (UID: \"05472180-97a7-40f5-9801-c39de1b24470\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:08:30.367007 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.366978    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05472180-97a7-40f5-9801-c39de1b24470-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq\" (UID: \"05472180-97a7-40f5-9801-c39de1b24470\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:08:30.367114 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.367003    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05472180-97a7-40f5-9801-c39de1b24470-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq\" (UID: \"05472180-97a7-40f5-9801-c39de1b24470\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:08:30.372650 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.372629    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqndd\" (UniqueName: \"kubernetes.io/projected/05472180-97a7-40f5-9801-c39de1b24470-kube-api-access-nqndd\") pod \"maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq\" (UID: \"05472180-97a7-40f5-9801-c39de1b24470\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:08:30.457609 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.457521    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:08:30.865266 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:08:30.865227    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05472180_97a7_40f5_9801_c39de1b24470.slice/crio-693fddb82a83e58bf2334f0b2547e272572470d990a0ff733c8bb3e92dc77e88 WatchSource:0}: Error finding container 693fddb82a83e58bf2334f0b2547e272572470d990a0ff733c8bb3e92dc77e88: Status 404 returned error can't find the container with id 693fddb82a83e58bf2334f0b2547e272572470d990a0ff733c8bb3e92dc77e88
Sep 11 12:08:30.870413 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:30.870390    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"]
Sep 11 12:08:31.618138 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.616997    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq" event={"ID":"05472180-97a7-40f5-9801-c39de1b24470","Type":"ContainerStarted","Data":"466c560c29347db9529c90065d80d9a5276f63792ef96c143e9f7a20d28d87d1"}
Sep 11 12:08:31.618629 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.618188    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:08:31.618629 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.618222    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq" event={"ID":"05472180-97a7-40f5-9801-c39de1b24470","Type":"ContainerStarted","Data":"693fddb82a83e58bf2334f0b2547e272572470d990a0ff733c8bb3e92dc77e88"}
Sep 11 12:08:31.640114 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.640064    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq" podStartSLOduration=1.346400035 podStartE2EDuration="1.640051199s" totalImagesPullingTime="293.651164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:08:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:08:31.637695401 +0000 UTC m=+1234.059563804" watchObservedRunningTime="2026-09-11 12:08:31.640051199 +0000 UTC m=+1234.061919634"
Sep 11 12:08:31.722218 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.722171    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"]
Sep 11 12:08:31.725882 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.725863    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.728420 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.728398    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-af081c\"/\"openshift-service-ca.crt\""
Sep 11 12:08:31.728579 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.728472    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-af081c\"/\"kube-root-ca.crt\""
Sep 11 12:08:31.728579 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.728554    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-af081c\"/\"auth-test-model-af081c-kserve-self-signed-certs\""
Sep 11 12:08:31.728799 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.728706    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-af081c\"/\"default-dockercfg-qbg9d\""
Sep 11 12:08:31.736541 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.736521    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"]
Sep 11 12:08:31.778472 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.778443    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-dshm\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.778640 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.778516    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-model-cache\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.778640 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.778545    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnfw7\" (UniqueName: \"kubernetes.io/projected/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-kube-api-access-lnfw7\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.778640 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.778582    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-home\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.778640 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.778619    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-tmp-dir\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.778814 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.778651    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-tls-certs\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.880057 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.879956    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-model-cache\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.880057 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.880000    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnfw7\" (UniqueName: \"kubernetes.io/projected/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-kube-api-access-lnfw7\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.880057 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.880038    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-home\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.880338 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.880065    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-tmp-dir\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.880338 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.880100    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-tls-certs\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.880338 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.880198    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-dshm\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.880506 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.880414    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-model-cache\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.880506 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.880450    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-tmp-dir\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.880506 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.880490    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-home\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.882431 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.882411    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-dshm\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.882645 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.882628    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-tls-certs\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:31.888656 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:31.888634    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnfw7\" (UniqueName: \"kubernetes.io/projected/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-kube-api-access-lnfw7\") pod \"auth-test-model-af081c-kserve-5f8c575d99-v7fzg\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") " pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:32.036374 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:32.036344    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:32.311159 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:32.311135    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"]
Sep 11 12:08:32.312742 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:08:32.312715    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb03e8780_6c4d_4b22_9af9_77367ef9bf7c.slice/crio-4e957c7f80fa02825ac98af3af3a80b134c973e649db2a0c813029900aa6affe WatchSource:0}: Error finding container 4e957c7f80fa02825ac98af3af3a80b134c973e649db2a0c813029900aa6affe: Status 404 returned error can't find the container with id 4e957c7f80fa02825ac98af3af3a80b134c973e649db2a0c813029900aa6affe
Sep 11 12:08:32.625610 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:32.625483    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg" event={"ID":"b03e8780-6c4d-4b22-9af9-77367ef9bf7c","Type":"ContainerStarted","Data":"4fc5d20d8c57cc25f7e6fe74caa83a7d7bfa94a5bf98bb27d3fbfb26390d46b7"}
Sep 11 12:08:32.625610 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:32.625532    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg" event={"ID":"b03e8780-6c4d-4b22-9af9-77367ef9bf7c","Type":"ContainerStarted","Data":"4e957c7f80fa02825ac98af3af3a80b134c973e649db2a0c813029900aa6affe"}
Sep 11 12:08:32.645743 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:32.645687    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg" podStartSLOduration=1.645669316 podStartE2EDuration="1.645669316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:08:32.643187146 +0000 UTC m=+1235.065055536" watchObservedRunningTime="2026-09-11 12:08:32.645669316 +0000 UTC m=+1235.067537700"
Sep 11 12:08:37.631315 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:37.631276    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:08:42.036936 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:42.036902    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:42.037422 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:42.036957    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:42.049238 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:42.049207    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:42.695884 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:42.695855    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:08:58.272859 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:08:58.272790    2569 scope.go:117] "RemoveContainer" containerID="3bdd0646be07d164c971e8a7aaaca4c81d503df89665e0cfaee35e1defb8fe96"
Sep 11 12:09:02.754105 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.754070    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"]
Sep 11 12:09:02.759016 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.758998    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.761695 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.761673    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d844cc\"/\"kube-root-ca.crt\""
Sep 11 12:09:02.761810 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.761706    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d844cc\"/\"openshift-service-ca.crt\""
Sep 11 12:09:02.762482 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.762467    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d844cc\"/\"default-dockercfg-fl7bt\""
Sep 11 12:09:02.762532 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.762498    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d844cc\"/\"auth-test-model-d844cc-kserve-self-signed-certs\""
Sep 11 12:09:02.769052 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.769029    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"]
Sep 11 12:09:02.862813 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.862780    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-model-cache\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.862813 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.862817    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-tmp-dir\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.863077 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.862844    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-dshm\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.863077 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.862918    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-home\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.863077 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.862968    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxzgw\" (UniqueName: \"kubernetes.io/projected/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-kube-api-access-fxzgw\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.863077 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.863045    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-tls-certs\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.964408 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.964371    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-model-cache\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.964408 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.964407    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-tmp-dir\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.964646 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.964432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-dshm\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.964646 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.964455    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-home\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.964646 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.964475    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxzgw\" (UniqueName: \"kubernetes.io/projected/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-kube-api-access-fxzgw\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.964646 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.964632    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-tls-certs\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.964878 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.964816    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-model-cache\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.964936 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.964888    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-tmp-dir\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.964936 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.964906    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-home\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.966732 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.966710    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-dshm\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.966998 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.966982    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-tls-certs\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:02.973145 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:02.973122    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxzgw\" (UniqueName: \"kubernetes.io/projected/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-kube-api-access-fxzgw\") pod \"auth-test-model-d844cc-kserve-75989b8c5f-rf64z\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:03.069298 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:03.069223    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:03.345537 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:03.345509    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"]
Sep 11 12:09:03.346476 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:09:03.346450    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8da3e1d3_ff66_4cb3_b44a_903ff1de58a8.slice/crio-d80dfb40ebe29f2d4f694ed23d47bcea17de7ab417e78754de0473673017b9ae WatchSource:0}: Error finding container d80dfb40ebe29f2d4f694ed23d47bcea17de7ab417e78754de0473673017b9ae: Status 404 returned error can't find the container with id d80dfb40ebe29f2d4f694ed23d47bcea17de7ab417e78754de0473673017b9ae
Sep 11 12:09:03.811340 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:03.811297    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z" event={"ID":"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8","Type":"ContainerStarted","Data":"8cdd315c2aed56cc1c5ba0e865e7a3066f50b769d80f8f69503b80f243a93eb9"}
Sep 11 12:09:03.811340 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:03.811343    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z" event={"ID":"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8","Type":"ContainerStarted","Data":"d80dfb40ebe29f2d4f694ed23d47bcea17de7ab417e78754de0473673017b9ae"}
Sep 11 12:09:03.832297 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:03.832241    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z" podStartSLOduration=1.832226215 podStartE2EDuration="1.832226215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:09:03.829433244 +0000 UTC m=+1266.251301624" watchObservedRunningTime="2026-09-11 12:09:03.832226215 +0000 UTC m=+1266.254094595"
Sep 11 12:09:13.069876 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:13.069841    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:13.070262 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:13.069897    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:13.082631 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:13.082575    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:13.883068 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:13.883039    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:09:48.086868 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.086838    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"]
Sep 11 12:09:48.090686 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.090665    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.093827 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.093809    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8d375e02\"/\"openshift-service-ca.crt\""
Sep 11 12:09:48.094824 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.094794    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8d375e02\"/\"kube-root-ca.crt\""
Sep 11 12:09:48.094925 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.094854    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8d375e02\"/\"e2e-lifecycle-model-8d375e02-kserve-self-signed-certs\""
Sep 11 12:09:48.095107 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.095090    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8d375e02\"/\"default-dockercfg-jdlnk\""
Sep 11 12:09:48.107534 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.107512    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"]
Sep 11 12:09:48.179867 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.179839    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsmpk\" (UniqueName: \"kubernetes.io/projected/6c394393-ce31-4fd0-be34-15ca0dbd538f-kube-api-access-wsmpk\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.179985 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.179916    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-tmp-dir\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.179985 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.179938    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-model-cache\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.179985 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.179982    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-home\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.180128 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.180096    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c394393-ce31-4fd0-be34-15ca0dbd538f-tls-certs\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.180175 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.180135    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-dshm\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.280576 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.280546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-tmp-dir\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.280745 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.280605    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-model-cache\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.280745 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.280638    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-home\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.280745 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.280661    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c394393-ce31-4fd0-be34-15ca0dbd538f-tls-certs\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.280745 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.280700    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-dshm\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.280975 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.280755    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsmpk\" (UniqueName: \"kubernetes.io/projected/6c394393-ce31-4fd0-be34-15ca0dbd538f-kube-api-access-wsmpk\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.281033 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.280990    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-tmp-dir\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.281080 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.281044    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-model-cache\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.281116 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.281081    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-home\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.282914 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.282885    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-dshm\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.283346 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.283327    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c394393-ce31-4fd0-be34-15ca0dbd538f-tls-certs\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.291085 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.291057    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsmpk\" (UniqueName: \"kubernetes.io/projected/6c394393-ce31-4fd0-be34-15ca0dbd538f-kube-api-access-wsmpk\") pod \"e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") " pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.400355 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.400281    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:48.691107 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:48.691075    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"]
Sep 11 12:09:48.692557 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:09:48.692535    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c394393_ce31_4fd0_be34_15ca0dbd538f.slice/crio-50e52d77b91412eb26120ca4301e4c657304e36997eade2fcbeda7a463126ef5 WatchSource:0}: Error finding container 50e52d77b91412eb26120ca4301e4c657304e36997eade2fcbeda7a463126ef5: Status 404 returned error can't find the container with id 50e52d77b91412eb26120ca4301e4c657304e36997eade2fcbeda7a463126ef5
Sep 11 12:09:49.087158 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:49.087126    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc" event={"ID":"6c394393-ce31-4fd0-be34-15ca0dbd538f","Type":"ContainerStarted","Data":"03701d7134a7a94e69dd096615f5e1577d18049fe25aaccc39a3ada2f36ce9f2"}
Sep 11 12:09:49.087663 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:49.087164    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc" event={"ID":"6c394393-ce31-4fd0-be34-15ca0dbd538f","Type":"ContainerStarted","Data":"50e52d77b91412eb26120ca4301e4c657304e36997eade2fcbeda7a463126ef5"}
Sep 11 12:09:58.295351 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:58.295308    2569 scope.go:117] "RemoveContainer" containerID="0e41355b0adf68670028725fa198df39551cbfd336e6df46114013cf23598ffb"
Sep 11 12:09:58.401425 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:58.401389    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:58.401612 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:58.401439    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:58.415165 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:58.415133    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:09:58.435699 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:58.435631    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc" podStartSLOduration=10.435610763 podStartE2EDuration="10.435610763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:09:49.131772305 +0000 UTC m=+1311.553640684" watchObservedRunningTime="2026-09-11 12:09:58.435610763 +0000 UTC m=+1320.857479144"
Sep 11 12:09:59.156286 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:09:59.156255    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:10:46.580928 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:10:46.580893    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"]
Sep 11 12:10:58.321664 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:10:58.321635    2569 scope.go:117] "RemoveContainer" containerID="131e6d89c702fc0e23509c18fcf301e47987885abf040b7eec9f92696c992ca6"
Sep 11 12:10:58.331026 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:10:58.331005    2569 scope.go:117] "RemoveContainer" containerID="b6b1749420337a75e712d30e37528221eb97e21847e666d9c9e3ce57d7e14a9c"
Sep 11 12:10:58.338825 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:10:58.338809    2569 scope.go:117] "RemoveContainer" containerID="ad6bde0321212c1f42fa2f187805cb45730219b90f1e99f9f176e4244cceea20"
Sep 11 12:11:01.600602 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.600538    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc" podUID="6c394393-ce31-4fd0-be34-15ca0dbd538f" containerName="main" containerID="cri-o://03701d7134a7a94e69dd096615f5e1577d18049fe25aaccc39a3ada2f36ce9f2" gracePeriod=45
Sep 11 12:11:01.848448 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.848419    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:11:01.877945 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.877877    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-tmp-dir\") pod \"6c394393-ce31-4fd0-be34-15ca0dbd538f\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") "
Sep 11 12:11:01.877945 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.877936    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-model-cache\") pod \"6c394393-ce31-4fd0-be34-15ca0dbd538f\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") "
Sep 11 12:11:01.878137 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.877984    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-dshm\") pod \"6c394393-ce31-4fd0-be34-15ca0dbd538f\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") "
Sep 11 12:11:01.878137 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.878018    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-home\") pod \"6c394393-ce31-4fd0-be34-15ca0dbd538f\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") "
Sep 11 12:11:01.878137 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.878055    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsmpk\" (UniqueName: \"kubernetes.io/projected/6c394393-ce31-4fd0-be34-15ca0dbd538f-kube-api-access-wsmpk\") pod \"6c394393-ce31-4fd0-be34-15ca0dbd538f\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") "
Sep 11 12:11:01.878137 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.878077    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c394393-ce31-4fd0-be34-15ca0dbd538f" (UID: "6c394393-ce31-4fd0-be34-15ca0dbd538f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:11:01.878356 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.878196    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c394393-ce31-4fd0-be34-15ca0dbd538f-tls-certs\") pod \"6c394393-ce31-4fd0-be34-15ca0dbd538f\" (UID: \"6c394393-ce31-4fd0-be34-15ca0dbd538f\") "
Sep 11 12:11:01.878356 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.878219    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c394393-ce31-4fd0-be34-15ca0dbd538f" (UID: "6c394393-ce31-4fd0-be34-15ca0dbd538f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:11:01.878356 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.878277    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-home" (OuterVolumeSpecName: "home") pod "6c394393-ce31-4fd0-be34-15ca0dbd538f" (UID: "6c394393-ce31-4fd0-be34-15ca0dbd538f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:11:01.878625 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.878567    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-model-cache\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:01.878625 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.878607    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-home\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:01.878625 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.878622    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-tmp-dir\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:01.880819 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.880791    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c394393-ce31-4fd0-be34-15ca0dbd538f-kube-api-access-wsmpk" (OuterVolumeSpecName: "kube-api-access-wsmpk") pod "6c394393-ce31-4fd0-be34-15ca0dbd538f" (UID: "6c394393-ce31-4fd0-be34-15ca0dbd538f"). InnerVolumeSpecName "kube-api-access-wsmpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:11:01.880991 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.880971    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c394393-ce31-4fd0-be34-15ca0dbd538f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c394393-ce31-4fd0-be34-15ca0dbd538f" (UID: "6c394393-ce31-4fd0-be34-15ca0dbd538f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:11:01.881058 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.880993    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-dshm" (OuterVolumeSpecName: "dshm") pod "6c394393-ce31-4fd0-be34-15ca0dbd538f" (UID: "6c394393-ce31-4fd0-be34-15ca0dbd538f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:11:01.979131 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.979106    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c394393-ce31-4fd0-be34-15ca0dbd538f-dshm\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:01.979131 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.979129    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsmpk\" (UniqueName: \"kubernetes.io/projected/6c394393-ce31-4fd0-be34-15ca0dbd538f-kube-api-access-wsmpk\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:01.979297 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:01.979140    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c394393-ce31-4fd0-be34-15ca0dbd538f-tls-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:02.516269 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:02.516233    2569 generic.go:356] "Generic (PLEG): container finished" podID="6c394393-ce31-4fd0-be34-15ca0dbd538f" containerID="03701d7134a7a94e69dd096615f5e1577d18049fe25aaccc39a3ada2f36ce9f2" exitCode=0
Sep 11 12:11:02.516428 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:02.516280    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"
Sep 11 12:11:02.516428 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:02.516317    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc" event={"ID":"6c394393-ce31-4fd0-be34-15ca0dbd538f","Type":"ContainerDied","Data":"03701d7134a7a94e69dd096615f5e1577d18049fe25aaccc39a3ada2f36ce9f2"}
Sep 11 12:11:02.516428 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:02.516364    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc" event={"ID":"6c394393-ce31-4fd0-be34-15ca0dbd538f","Type":"ContainerDied","Data":"50e52d77b91412eb26120ca4301e4c657304e36997eade2fcbeda7a463126ef5"}
Sep 11 12:11:02.516428 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:02.516382    2569 scope.go:117] "RemoveContainer" containerID="03701d7134a7a94e69dd096615f5e1577d18049fe25aaccc39a3ada2f36ce9f2"
Sep 11 12:11:02.525563 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:02.525547    2569 scope.go:117] "RemoveContainer" containerID="03701d7134a7a94e69dd096615f5e1577d18049fe25aaccc39a3ada2f36ce9f2"
Sep 11 12:11:02.525890 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:11:02.525869    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03701d7134a7a94e69dd096615f5e1577d18049fe25aaccc39a3ada2f36ce9f2\": container with ID starting with 03701d7134a7a94e69dd096615f5e1577d18049fe25aaccc39a3ada2f36ce9f2 not found: ID does not exist" containerID="03701d7134a7a94e69dd096615f5e1577d18049fe25aaccc39a3ada2f36ce9f2"
Sep 11 12:11:02.525943 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:02.525898    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03701d7134a7a94e69dd096615f5e1577d18049fe25aaccc39a3ada2f36ce9f2"} err="failed to get container status \"03701d7134a7a94e69dd096615f5e1577d18049fe25aaccc39a3ada2f36ce9f2\": rpc error: code = NotFound desc = could not find container \"03701d7134a7a94e69dd096615f5e1577d18049fe25aaccc39a3ada2f36ce9f2\": container with ID starting with 03701d7134a7a94e69dd096615f5e1577d18049fe25aaccc39a3ada2f36ce9f2 not found: ID does not exist"
Sep 11 12:11:02.536816 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:02.536294    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"]
Sep 11 12:11:02.542488 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:02.542452    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-8d375e02/e2e-lifecycle-model-8d375e02-kserve-77dc6cd4dc-jmnhc"]
Sep 11 12:11:04.070264 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:04.070229    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c394393-ce31-4fd0-be34-15ca0dbd538f" path="/var/lib/kubelet/pods/6c394393-ce31-4fd0-be34-15ca0dbd538f/volumes"
Sep 11 12:11:21.020532 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:21.020499    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"]
Sep 11 12:11:22.035878 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:22.035845    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"]
Sep 11 12:11:36.039303 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.039253    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg" podUID="b03e8780-6c4d-4b22-9af9-77367ef9bf7c" containerName="main" containerID="cri-o://4fc5d20d8c57cc25f7e6fe74caa83a7d7bfa94a5bf98bb27d3fbfb26390d46b7" gracePeriod=45
Sep 11 12:11:36.281310 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.281287    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:11:36.389194 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.389129    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-model-cache\") pod \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") "
Sep 11 12:11:36.389194 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.389163    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-dshm\") pod \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") "
Sep 11 12:11:36.389194 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.389196    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-tmp-dir\") pod \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") "
Sep 11 12:11:36.389455 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.389292    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-tls-certs\") pod \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") "
Sep 11 12:11:36.389455 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.389318    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnfw7\" (UniqueName: \"kubernetes.io/projected/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-kube-api-access-lnfw7\") pod \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") "
Sep 11 12:11:36.389455 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.389351    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-home\") pod \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\" (UID: \"b03e8780-6c4d-4b22-9af9-77367ef9bf7c\") "
Sep 11 12:11:36.389455 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.389421    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "b03e8780-6c4d-4b22-9af9-77367ef9bf7c" (UID: "b03e8780-6c4d-4b22-9af9-77367ef9bf7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:11:36.389663 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.389451    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b03e8780-6c4d-4b22-9af9-77367ef9bf7c" (UID: "b03e8780-6c4d-4b22-9af9-77367ef9bf7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:11:36.389707 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.389655    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-home" (OuterVolumeSpecName: "home") pod "b03e8780-6c4d-4b22-9af9-77367ef9bf7c" (UID: "b03e8780-6c4d-4b22-9af9-77367ef9bf7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:11:36.389707 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.389672    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-model-cache\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:36.389707 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.389691    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-tmp-dir\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:36.391414 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.391383    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-dshm" (OuterVolumeSpecName: "dshm") pod "b03e8780-6c4d-4b22-9af9-77367ef9bf7c" (UID: "b03e8780-6c4d-4b22-9af9-77367ef9bf7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:11:36.391520 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.391425    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b03e8780-6c4d-4b22-9af9-77367ef9bf7c" (UID: "b03e8780-6c4d-4b22-9af9-77367ef9bf7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:11:36.391520 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.391437    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-kube-api-access-lnfw7" (OuterVolumeSpecName: "kube-api-access-lnfw7") pod "b03e8780-6c4d-4b22-9af9-77367ef9bf7c" (UID: "b03e8780-6c4d-4b22-9af9-77367ef9bf7c"). InnerVolumeSpecName "kube-api-access-lnfw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:11:36.490273 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.490245    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-tls-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:36.490273 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.490271    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnfw7\" (UniqueName: \"kubernetes.io/projected/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-kube-api-access-lnfw7\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:36.490410 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.490282    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-home\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:36.490410 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.490291    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03e8780-6c4d-4b22-9af9-77367ef9bf7c-dshm\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:36.717382 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.717300    2569 generic.go:356] "Generic (PLEG): container finished" podID="b03e8780-6c4d-4b22-9af9-77367ef9bf7c" containerID="4fc5d20d8c57cc25f7e6fe74caa83a7d7bfa94a5bf98bb27d3fbfb26390d46b7" exitCode=0
Sep 11 12:11:36.717382 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.717341    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"
Sep 11 12:11:36.717540 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.717387    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg" event={"ID":"b03e8780-6c4d-4b22-9af9-77367ef9bf7c","Type":"ContainerDied","Data":"4fc5d20d8c57cc25f7e6fe74caa83a7d7bfa94a5bf98bb27d3fbfb26390d46b7"}
Sep 11 12:11:36.717540 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.717431    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg" event={"ID":"b03e8780-6c4d-4b22-9af9-77367ef9bf7c","Type":"ContainerDied","Data":"4e957c7f80fa02825ac98af3af3a80b134c973e649db2a0c813029900aa6affe"}
Sep 11 12:11:36.717540 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.717453    2569 scope.go:117] "RemoveContainer" containerID="4fc5d20d8c57cc25f7e6fe74caa83a7d7bfa94a5bf98bb27d3fbfb26390d46b7"
Sep 11 12:11:36.727420 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.727404    2569 scope.go:117] "RemoveContainer" containerID="4fc5d20d8c57cc25f7e6fe74caa83a7d7bfa94a5bf98bb27d3fbfb26390d46b7"
Sep 11 12:11:36.727701 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:11:36.727678    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fc5d20d8c57cc25f7e6fe74caa83a7d7bfa94a5bf98bb27d3fbfb26390d46b7\": container with ID starting with 4fc5d20d8c57cc25f7e6fe74caa83a7d7bfa94a5bf98bb27d3fbfb26390d46b7 not found: ID does not exist" containerID="4fc5d20d8c57cc25f7e6fe74caa83a7d7bfa94a5bf98bb27d3fbfb26390d46b7"
Sep 11 12:11:36.727757 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.727712    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fc5d20d8c57cc25f7e6fe74caa83a7d7bfa94a5bf98bb27d3fbfb26390d46b7"} err="failed to get container status \"4fc5d20d8c57cc25f7e6fe74caa83a7d7bfa94a5bf98bb27d3fbfb26390d46b7\": rpc error: code = NotFound desc = could not find container \"4fc5d20d8c57cc25f7e6fe74caa83a7d7bfa94a5bf98bb27d3fbfb26390d46b7\": container with ID starting with 4fc5d20d8c57cc25f7e6fe74caa83a7d7bfa94a5bf98bb27d3fbfb26390d46b7 not found: ID does not exist"
Sep 11 12:11:36.741551 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.741529    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"]
Sep 11 12:11:36.745951 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:36.745931    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af081c/auth-test-model-af081c-kserve-5f8c575d99-v7fzg"]
Sep 11 12:11:37.062335 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.062298    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z" podUID="8da3e1d3-ff66-4cb3-b44a-903ff1de58a8" containerName="main" containerID="cri-o://8cdd315c2aed56cc1c5ba0e865e7a3066f50b769d80f8f69503b80f243a93eb9" gracePeriod=45
Sep 11 12:11:37.315947 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.315890    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:11:37.499003 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.498973    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-tmp-dir\") pod \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") "
Sep 11 12:11:37.499147 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.499017    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-model-cache\") pod \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") "
Sep 11 12:11:37.499147 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.499056    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxzgw\" (UniqueName: \"kubernetes.io/projected/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-kube-api-access-fxzgw\") pod \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") "
Sep 11 12:11:37.499147 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.499072    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-tls-certs\") pod \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") "
Sep 11 12:11:37.499147 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.499089    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-dshm\") pod \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") "
Sep 11 12:11:37.499364 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.499169    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-home\") pod \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\" (UID: \"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8\") "
Sep 11 12:11:37.499364 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.499305    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8da3e1d3-ff66-4cb3-b44a-903ff1de58a8" (UID: "8da3e1d3-ff66-4cb3-b44a-903ff1de58a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:11:37.499479 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.499416    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "8da3e1d3-ff66-4cb3-b44a-903ff1de58a8" (UID: "8da3e1d3-ff66-4cb3-b44a-903ff1de58a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:11:37.499535 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.499474    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-home" (OuterVolumeSpecName: "home") pod "8da3e1d3-ff66-4cb3-b44a-903ff1de58a8" (UID: "8da3e1d3-ff66-4cb3-b44a-903ff1de58a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:11:37.499610 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.499555    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-tmp-dir\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:37.499610 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.499571    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-model-cache\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:37.499610 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.499579    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-home\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:37.501497 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.501470    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-dshm" (OuterVolumeSpecName: "dshm") pod "8da3e1d3-ff66-4cb3-b44a-903ff1de58a8" (UID: "8da3e1d3-ff66-4cb3-b44a-903ff1de58a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:11:37.501630 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.501496    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-kube-api-access-fxzgw" (OuterVolumeSpecName: "kube-api-access-fxzgw") pod "8da3e1d3-ff66-4cb3-b44a-903ff1de58a8" (UID: "8da3e1d3-ff66-4cb3-b44a-903ff1de58a8"). InnerVolumeSpecName "kube-api-access-fxzgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:11:37.501630 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.501473    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8da3e1d3-ff66-4cb3-b44a-903ff1de58a8" (UID: "8da3e1d3-ff66-4cb3-b44a-903ff1de58a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:11:37.600003 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.599950    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxzgw\" (UniqueName: \"kubernetes.io/projected/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-kube-api-access-fxzgw\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:37.600003 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.599971    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-tls-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:37.600003 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.599979    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8-dshm\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:11:37.725748 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.725719    2569 generic.go:356] "Generic (PLEG): container finished" podID="8da3e1d3-ff66-4cb3-b44a-903ff1de58a8" containerID="8cdd315c2aed56cc1c5ba0e865e7a3066f50b769d80f8f69503b80f243a93eb9" exitCode=0
Sep 11 12:11:37.725876 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.725762    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z" event={"ID":"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8","Type":"ContainerDied","Data":"8cdd315c2aed56cc1c5ba0e865e7a3066f50b769d80f8f69503b80f243a93eb9"}
Sep 11 12:11:37.725876 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.725794    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z" event={"ID":"8da3e1d3-ff66-4cb3-b44a-903ff1de58a8","Type":"ContainerDied","Data":"d80dfb40ebe29f2d4f694ed23d47bcea17de7ab417e78754de0473673017b9ae"}
Sep 11 12:11:37.725876 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.725811    2569 scope.go:117] "RemoveContainer" containerID="8cdd315c2aed56cc1c5ba0e865e7a3066f50b769d80f8f69503b80f243a93eb9"
Sep 11 12:11:37.725876 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.725767    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"
Sep 11 12:11:37.735831 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.735813    2569 scope.go:117] "RemoveContainer" containerID="8cdd315c2aed56cc1c5ba0e865e7a3066f50b769d80f8f69503b80f243a93eb9"
Sep 11 12:11:37.736115 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:11:37.736094    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cdd315c2aed56cc1c5ba0e865e7a3066f50b769d80f8f69503b80f243a93eb9\": container with ID starting with 8cdd315c2aed56cc1c5ba0e865e7a3066f50b769d80f8f69503b80f243a93eb9 not found: ID does not exist" containerID="8cdd315c2aed56cc1c5ba0e865e7a3066f50b769d80f8f69503b80f243a93eb9"
Sep 11 12:11:37.736170 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.736124    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cdd315c2aed56cc1c5ba0e865e7a3066f50b769d80f8f69503b80f243a93eb9"} err="failed to get container status \"8cdd315c2aed56cc1c5ba0e865e7a3066f50b769d80f8f69503b80f243a93eb9\": rpc error: code = NotFound desc = could not find container \"8cdd315c2aed56cc1c5ba0e865e7a3066f50b769d80f8f69503b80f243a93eb9\": container with ID starting with 8cdd315c2aed56cc1c5ba0e865e7a3066f50b769d80f8f69503b80f243a93eb9 not found: ID does not exist"
Sep 11 12:11:37.764094 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.764074    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"]
Sep 11 12:11:37.783541 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:37.783516    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/auth-test-model-d844cc-kserve-75989b8c5f-rf64z"]
Sep 11 12:11:38.070708 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:38.070678    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8da3e1d3-ff66-4cb3-b44a-903ff1de58a8" path="/var/lib/kubelet/pods/8da3e1d3-ff66-4cb3-b44a-903ff1de58a8/volumes"
Sep 11 12:11:38.071091 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:38.071077    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b03e8780-6c4d-4b22-9af9-77367ef9bf7c" path="/var/lib/kubelet/pods/b03e8780-6c4d-4b22-9af9-77367ef9bf7c/volumes"
Sep 11 12:11:45.559403 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.559371    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"]
Sep 11 12:11:45.559870 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.559851    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c394393-ce31-4fd0-be34-15ca0dbd538f" containerName="main"
Sep 11 12:11:45.559870 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.559871    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c394393-ce31-4fd0-be34-15ca0dbd538f" containerName="main"
Sep 11 12:11:45.559965 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.559891    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8da3e1d3-ff66-4cb3-b44a-903ff1de58a8" containerName="main"
Sep 11 12:11:45.559965 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.559896    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8da3e1d3-ff66-4cb3-b44a-903ff1de58a8" containerName="main"
Sep 11 12:11:45.559965 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.559906    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03e8780-6c4d-4b22-9af9-77367ef9bf7c" containerName="main"
Sep 11 12:11:45.559965 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.559912    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03e8780-6c4d-4b22-9af9-77367ef9bf7c" containerName="main"
Sep 11 12:11:45.560115 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.559988    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b03e8780-6c4d-4b22-9af9-77367ef9bf7c" containerName="main"
Sep 11 12:11:45.560115 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.559998    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8da3e1d3-ff66-4cb3-b44a-903ff1de58a8" containerName="main"
Sep 11 12:11:45.560115 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.560004    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c394393-ce31-4fd0-be34-15ca0dbd538f" containerName="main"
Sep 11 12:11:45.564485 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.564461    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.568254 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.568214    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d844cc\"/\"openshift-service-ca.crt\""
Sep 11 12:11:45.568254 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.568230    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d844cc\"/\"sub-test-model-d844cc-kserve-self-signed-certs\""
Sep 11 12:11:45.568456 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.568272    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d844cc\"/\"kube-root-ca.crt\""
Sep 11 12:11:45.568456 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.568339    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d844cc\"/\"default-dockercfg-fl7bt\""
Sep 11 12:11:45.568711 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.568688    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4pj7\" (UniqueName: \"kubernetes.io/projected/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-kube-api-access-q4pj7\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.568810 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.568761    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-tmp-dir\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.568878 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.568813    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-model-cache\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.568945 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.568888    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-tls-certs\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.568945 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.568927    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-dshm\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.569055 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.568986    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-home\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.575750 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.575730    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"]
Sep 11 12:11:45.670053 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.670024    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-dshm\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.670172 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.670070    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-home\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.670172 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.670113    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4pj7\" (UniqueName: \"kubernetes.io/projected/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-kube-api-access-q4pj7\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.670172 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.670166    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-tmp-dir\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.670349 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.670212    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-model-cache\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.670349 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.670267    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-tls-certs\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.670503 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.670474    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-home\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.670640 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.670521    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-tmp-dir\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.670806 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.670785    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-model-cache\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.672334 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.672313    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-dshm\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.672739 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.672722    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-tls-certs\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.679716 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.679690    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4pj7\" (UniqueName: \"kubernetes.io/projected/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-kube-api-access-q4pj7\") pod \"sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:45.875198 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:45.875121    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:46.159974 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:46.159946    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"]
Sep 11 12:11:46.161334 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:11:46.161297    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4db8d0ed_5ed2_4e2f_b3af_ba12172819ce.slice/crio-f85b8dcc70b76b96df2024b8617d5a10fb1e50236fdbb91be965d927d8c9cb01 WatchSource:0}: Error finding container f85b8dcc70b76b96df2024b8617d5a10fb1e50236fdbb91be965d927d8c9cb01: Status 404 returned error can't find the container with id f85b8dcc70b76b96df2024b8617d5a10fb1e50236fdbb91be965d927d8c9cb01
Sep 11 12:11:46.779219 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:46.779183    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k" event={"ID":"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce","Type":"ContainerStarted","Data":"9622e41db8ec6f12dd8f4abf928f2ac5f5aaf36001a16f944b9fb7c9a9914aa8"}
Sep 11 12:11:46.779709 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:46.779229    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k" event={"ID":"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce","Type":"ContainerStarted","Data":"f85b8dcc70b76b96df2024b8617d5a10fb1e50236fdbb91be965d927d8c9cb01"}
Sep 11 12:11:46.805428 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:46.805387    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k" podStartSLOduration=1.805372755 podStartE2EDuration="1.805372755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:11:46.804444467 +0000 UTC m=+1429.226312825" watchObservedRunningTime="2026-09-11 12:11:46.805372755 +0000 UTC m=+1429.227241134"
Sep 11 12:11:55.875966 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:55.875890    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:55.875966 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:55.875941    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:55.888330 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:55.888306    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:11:56.849128 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:11:56.849100    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:12:25.007473 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:25.007440    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7465-32c781"]
Sep 11 12:12:25.011054 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:25.011030    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7465-32c781"
Sep 11 12:12:25.013913 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:25.013892    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wkgmv\""
Sep 11 12:12:25.020064 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:25.020044    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7465-32c781"]
Sep 11 12:12:25.125335 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:25.125315    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pkl7\" (UniqueName: \"kubernetes.io/projected/722db745-be94-472c-ae65-e7fd9d694915-kube-api-access-5pkl7\") pod \"test-curl-7465-32c781\" (UID: \"722db745-be94-472c-ae65-e7fd9d694915\") " pod="opendatahub/test-curl-7465-32c781"
Sep 11 12:12:25.226735 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:25.226710    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pkl7\" (UniqueName: \"kubernetes.io/projected/722db745-be94-472c-ae65-e7fd9d694915-kube-api-access-5pkl7\") pod \"test-curl-7465-32c781\" (UID: \"722db745-be94-472c-ae65-e7fd9d694915\") " pod="opendatahub/test-curl-7465-32c781"
Sep 11 12:12:25.235775 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:25.235758    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pkl7\" (UniqueName: \"kubernetes.io/projected/722db745-be94-472c-ae65-e7fd9d694915-kube-api-access-5pkl7\") pod \"test-curl-7465-32c781\" (UID: \"722db745-be94-472c-ae65-e7fd9d694915\") " pod="opendatahub/test-curl-7465-32c781"
Sep 11 12:12:25.320860 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:25.320809    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7465-32c781"
Sep 11 12:12:25.600451 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:25.600423    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7465-32c781"]
Sep 11 12:12:25.601643 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:12:25.601617    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod722db745_be94_472c_ae65_e7fd9d694915.slice/crio-aae23c13e2bae042b06cda1b6d0e3c6a8c62ad1e4c6b8b593313afc1ee57617b WatchSource:0}: Error finding container aae23c13e2bae042b06cda1b6d0e3c6a8c62ad1e4c6b8b593313afc1ee57617b: Status 404 returned error can't find the container with id aae23c13e2bae042b06cda1b6d0e3c6a8c62ad1e4c6b8b593313afc1ee57617b
Sep 11 12:12:26.009874 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:26.009841    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7465-32c781" event={"ID":"722db745-be94-472c-ae65-e7fd9d694915","Type":"ContainerStarted","Data":"3a5cb322823bc840a6abe1a2020192b17e31dd70a00130ad2612ada237b84cff"}
Sep 11 12:12:26.010263 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:26.009879    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7465-32c781" event={"ID":"722db745-be94-472c-ae65-e7fd9d694915","Type":"ContainerStarted","Data":"aae23c13e2bae042b06cda1b6d0e3c6a8c62ad1e4c6b8b593313afc1ee57617b"}
Sep 11 12:12:26.027556 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:26.027510    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7465-32c781" podStartSLOduration=2.027496467 podStartE2EDuration="2.027496467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:12:26.027335003 +0000 UTC m=+1468.449203378" watchObservedRunningTime="2026-09-11 12:12:26.027496467 +0000 UTC m=+1468.449364846"
Sep 11 12:12:26.614255 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:26.614198    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7465-32c781"]
Sep 11 12:12:26.617564 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:26.617532    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7465-32c781"]
Sep 11 12:12:27.016214 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:12:27.016178    2569 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7465-32c781" container="test-curl-7465-32c781"
Sep 11 12:12:28.022194 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:28.022132    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7465-32c781" podUID="722db745-be94-472c-ae65-e7fd9d694915" containerName="test-curl-7465-32c781" containerID="cri-o://3a5cb322823bc840a6abe1a2020192b17e31dd70a00130ad2612ada237b84cff" gracePeriod=2
Sep 11 12:12:28.024294 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:12:28.024264    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7465-32c781\" is forbidden: User \"system:node:ip-10-0-143-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-220.ec2.internal' and this object" podUID="722db745-be94-472c-ae65-e7fd9d694915" pod="opendatahub/test-curl-7465-32c781"
Sep 11 12:12:28.069080 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:12:28.069047    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7465-32c781\" is forbidden: User \"system:node:ip-10-0-143-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-220.ec2.internal' and this object" podUID="722db745-be94-472c-ae65-e7fd9d694915" pod="opendatahub/test-curl-7465-32c781"
Sep 11 12:12:30.168919 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:30.168896    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7465-32c781"
Sep 11 12:12:30.272288 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:30.272265    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pkl7\" (UniqueName: \"kubernetes.io/projected/722db745-be94-472c-ae65-e7fd9d694915-kube-api-access-5pkl7\") pod \"722db745-be94-472c-ae65-e7fd9d694915\" (UID: \"722db745-be94-472c-ae65-e7fd9d694915\") "
Sep 11 12:12:30.274165 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:30.274138    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722db745-be94-472c-ae65-e7fd9d694915-kube-api-access-5pkl7" (OuterVolumeSpecName: "kube-api-access-5pkl7") pod "722db745-be94-472c-ae65-e7fd9d694915" (UID: "722db745-be94-472c-ae65-e7fd9d694915"). InnerVolumeSpecName "kube-api-access-5pkl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:12:30.373151 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:30.373124    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pkl7\" (UniqueName: \"kubernetes.io/projected/722db745-be94-472c-ae65-e7fd9d694915-kube-api-access-5pkl7\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:12:30.536322 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:30.536242    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"]
Sep 11 12:12:31.042630 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.042570    2569 generic.go:356] "Generic (PLEG): container finished" podID="722db745-be94-472c-ae65-e7fd9d694915" containerID="3a5cb322823bc840a6abe1a2020192b17e31dd70a00130ad2612ada237b84cff" exitCode=137
Sep 11 12:12:31.042630 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.042624    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7465-32c781"
Sep 11 12:12:31.042833 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.042624    2569 scope.go:117] "RemoveContainer" containerID="3a5cb322823bc840a6abe1a2020192b17e31dd70a00130ad2612ada237b84cff"
Sep 11 12:12:31.052119 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.052103    2569 scope.go:117] "RemoveContainer" containerID="3a5cb322823bc840a6abe1a2020192b17e31dd70a00130ad2612ada237b84cff"
Sep 11 12:12:31.052373 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:12:31.052354    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5cb322823bc840a6abe1a2020192b17e31dd70a00130ad2612ada237b84cff\": container with ID starting with 3a5cb322823bc840a6abe1a2020192b17e31dd70a00130ad2612ada237b84cff not found: ID does not exist" containerID="3a5cb322823bc840a6abe1a2020192b17e31dd70a00130ad2612ada237b84cff"
Sep 11 12:12:31.052434 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.052382    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5cb322823bc840a6abe1a2020192b17e31dd70a00130ad2612ada237b84cff"} err="failed to get container status \"3a5cb322823bc840a6abe1a2020192b17e31dd70a00130ad2612ada237b84cff\": rpc error: code = NotFound desc = could not find container \"3a5cb322823bc840a6abe1a2020192b17e31dd70a00130ad2612ada237b84cff\": container with ID starting with 3a5cb322823bc840a6abe1a2020192b17e31dd70a00130ad2612ada237b84cff not found: ID does not exist"
Sep 11 12:12:31.052659 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:12:31.052641    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7465-32c781\" is forbidden: User \"system:node:ip-10-0-143-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-220.ec2.internal' and this object" podUID="722db745-be94-472c-ae65-e7fd9d694915" pod="opendatahub/test-curl-7465-32c781"
Sep 11 12:12:31.324870 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.324787    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"]
Sep 11 12:12:31.325296 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.325279    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="722db745-be94-472c-ae65-e7fd9d694915" containerName="test-curl-7465-32c781"
Sep 11 12:12:31.325343 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.325299    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="722db745-be94-472c-ae65-e7fd9d694915" containerName="test-curl-7465-32c781"
Sep 11 12:12:31.325379 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.325370    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="722db745-be94-472c-ae65-e7fd9d694915" containerName="test-curl-7465-32c781"
Sep 11 12:12:31.329577 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.329559    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:12:31.332137 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.332117    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fb3c56-serving-cert\""
Sep 11 12:12:31.343156 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.343138    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"]
Sep 11 12:12:31.360339 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:12:31.360308    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7465-32c781\" is forbidden: User \"system:node:ip-10-0-143-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-220.ec2.internal' and this object" podUID="722db745-be94-472c-ae65-e7fd9d694915" pod="opendatahub/test-curl-7465-32c781"
Sep 11 12:12:31.383523 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.383495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc9dee78-9436-4c0a-918d-91660299eb37-maas-api-tls\") pod \"maas-api-e2e-api-a-fb3c56-59b5ff568-894fl\" (UID: \"cc9dee78-9436-4c0a-918d-91660299eb37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:12:31.383662 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.383629    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc9dee78-9436-4c0a-918d-91660299eb37-metrics-certs\") pod \"maas-api-e2e-api-a-fb3c56-59b5ff568-894fl\" (UID: \"cc9dee78-9436-4c0a-918d-91660299eb37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:12:31.383737 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.383671    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hth6\" (UniqueName: \"kubernetes.io/projected/cc9dee78-9436-4c0a-918d-91660299eb37-kube-api-access-8hth6\") pod \"maas-api-e2e-api-a-fb3c56-59b5ff568-894fl\" (UID: \"cc9dee78-9436-4c0a-918d-91660299eb37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:12:31.383796 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.383749    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc9dee78-9436-4c0a-918d-91660299eb37-openshift-service-ca\") pod \"maas-api-e2e-api-a-fb3c56-59b5ff568-894fl\" (UID: \"cc9dee78-9436-4c0a-918d-91660299eb37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:12:31.484553 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.484528    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hth6\" (UniqueName: \"kubernetes.io/projected/cc9dee78-9436-4c0a-918d-91660299eb37-kube-api-access-8hth6\") pod \"maas-api-e2e-api-a-fb3c56-59b5ff568-894fl\" (UID: \"cc9dee78-9436-4c0a-918d-91660299eb37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:12:31.484676 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.484571    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc9dee78-9436-4c0a-918d-91660299eb37-openshift-service-ca\") pod \"maas-api-e2e-api-a-fb3c56-59b5ff568-894fl\" (UID: \"cc9dee78-9436-4c0a-918d-91660299eb37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:12:31.484676 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.484657    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc9dee78-9436-4c0a-918d-91660299eb37-maas-api-tls\") pod \"maas-api-e2e-api-a-fb3c56-59b5ff568-894fl\" (UID: \"cc9dee78-9436-4c0a-918d-91660299eb37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:12:31.484756 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.484685    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc9dee78-9436-4c0a-918d-91660299eb37-metrics-certs\") pod \"maas-api-e2e-api-a-fb3c56-59b5ff568-894fl\" (UID: \"cc9dee78-9436-4c0a-918d-91660299eb37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:12:31.485339 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.485315    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc9dee78-9436-4c0a-918d-91660299eb37-openshift-service-ca\") pod \"maas-api-e2e-api-a-fb3c56-59b5ff568-894fl\" (UID: \"cc9dee78-9436-4c0a-918d-91660299eb37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:12:31.487118 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.487098    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc9dee78-9436-4c0a-918d-91660299eb37-metrics-certs\") pod \"maas-api-e2e-api-a-fb3c56-59b5ff568-894fl\" (UID: \"cc9dee78-9436-4c0a-918d-91660299eb37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:12:31.487209 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.487148    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc9dee78-9436-4c0a-918d-91660299eb37-maas-api-tls\") pod \"maas-api-e2e-api-a-fb3c56-59b5ff568-894fl\" (UID: \"cc9dee78-9436-4c0a-918d-91660299eb37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:12:31.493631 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.493570    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hth6\" (UniqueName: \"kubernetes.io/projected/cc9dee78-9436-4c0a-918d-91660299eb37-kube-api-access-8hth6\") pod \"maas-api-e2e-api-a-fb3c56-59b5ff568-894fl\" (UID: \"cc9dee78-9436-4c0a-918d-91660299eb37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:12:31.639330 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.639250    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:12:31.922121 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.922097    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"]
Sep 11 12:12:31.923781 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:12:31.923748    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9dee78_9436_4c0a_918d_91660299eb37.slice/crio-49d1e0be2801a06b3bd1f23f836b80182acf71fa819c9a3c2733fac8ba42f5e1 WatchSource:0}: Error finding container 49d1e0be2801a06b3bd1f23f836b80182acf71fa819c9a3c2733fac8ba42f5e1: Status 404 returned error can't find the container with id 49d1e0be2801a06b3bd1f23f836b80182acf71fa819c9a3c2733fac8ba42f5e1
Sep 11 12:12:31.925185 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:31.925165    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:12:32.049792 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:32.049758    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl" event={"ID":"cc9dee78-9436-4c0a-918d-91660299eb37","Type":"ContainerStarted","Data":"49d1e0be2801a06b3bd1f23f836b80182acf71fa819c9a3c2733fac8ba42f5e1"}
Sep 11 12:12:32.069947 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:32.069920    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="722db745-be94-472c-ae65-e7fd9d694915" path="/var/lib/kubelet/pods/722db745-be94-472c-ae65-e7fd9d694915/volumes"
Sep 11 12:12:33.058324 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:33.058282    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl" event={"ID":"cc9dee78-9436-4c0a-918d-91660299eb37","Type":"ContainerStarted","Data":"0da718abd7d6c1d38c2c7a20cca857c20234bd3e25e2a25ad7ef979673a9e4f8"}
Sep 11 12:12:33.058692 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:33.058385    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:12:33.076255 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:33.076210    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl" podStartSLOduration=1.549423478 podStartE2EDuration="2.076194223s" totalImagesPullingTime="526.770745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:12:33.075156995 +0000 UTC m=+1475.497025402" watchObservedRunningTime="2026-09-11 12:12:33.076194223 +0000 UTC m=+1475.498062677"
Sep 11 12:12:39.069708 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:39.069676    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:12:45.555070 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:45.555032    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k" podUID="4db8d0ed-5ed2-4e2f-b3af-ba12172819ce" containerName="main" containerID="cri-o://9622e41db8ec6f12dd8f4abf928f2ac5f5aaf36001a16f944b9fb7c9a9914aa8" gracePeriod=45
Sep 11 12:12:45.812617 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:45.812550    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:12:45.916730 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:45.916700    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4pj7\" (UniqueName: \"kubernetes.io/projected/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-kube-api-access-q4pj7\") pod \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") "
Sep 11 12:12:45.916879 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:45.916750    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-tmp-dir\") pod \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") "
Sep 11 12:12:45.916947 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:45.916929    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-dshm\") pod \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") "
Sep 11 12:12:45.916984 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:45.916962    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-model-cache\") pod \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") "
Sep 11 12:12:45.916984 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:45.916966    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4db8d0ed-5ed2-4e2f-b3af-ba12172819ce" (UID: "4db8d0ed-5ed2-4e2f-b3af-ba12172819ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:12:45.917063 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:45.917003    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-tls-certs\") pod \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") "
Sep 11 12:12:45.917063 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:45.917047    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-home\") pod \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\" (UID: \"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce\") "
Sep 11 12:12:45.917196 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:45.917171    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "4db8d0ed-5ed2-4e2f-b3af-ba12172819ce" (UID: "4db8d0ed-5ed2-4e2f-b3af-ba12172819ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:12:45.917376 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:45.917352    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-home" (OuterVolumeSpecName: "home") pod "4db8d0ed-5ed2-4e2f-b3af-ba12172819ce" (UID: "4db8d0ed-5ed2-4e2f-b3af-ba12172819ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:12:45.917460 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:45.917445    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-model-cache\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:12:45.917460 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:45.917457    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-home\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:12:45.917577 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:45.917466    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-tmp-dir\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:12:45.919007 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:45.918980    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-dshm" (OuterVolumeSpecName: "dshm") pod "4db8d0ed-5ed2-4e2f-b3af-ba12172819ce" (UID: "4db8d0ed-5ed2-4e2f-b3af-ba12172819ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:12:45.919007 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:45.918992    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4db8d0ed-5ed2-4e2f-b3af-ba12172819ce" (UID: "4db8d0ed-5ed2-4e2f-b3af-ba12172819ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:12:45.919007 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:45.918997    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-kube-api-access-q4pj7" (OuterVolumeSpecName: "kube-api-access-q4pj7") pod "4db8d0ed-5ed2-4e2f-b3af-ba12172819ce" (UID: "4db8d0ed-5ed2-4e2f-b3af-ba12172819ce"). InnerVolumeSpecName "kube-api-access-q4pj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:12:46.018799 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:46.018772    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-dshm\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:12:46.018799 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:46.018794    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-tls-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:12:46.018799 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:46.018804    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4pj7\" (UniqueName: \"kubernetes.io/projected/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce-kube-api-access-q4pj7\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:12:46.131448 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:46.131419    2569 generic.go:356] "Generic (PLEG): container finished" podID="4db8d0ed-5ed2-4e2f-b3af-ba12172819ce" containerID="9622e41db8ec6f12dd8f4abf928f2ac5f5aaf36001a16f944b9fb7c9a9914aa8" exitCode=0
Sep 11 12:12:46.131566 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:46.131462    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"
Sep 11 12:12:46.131566 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:46.131506    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k" event={"ID":"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce","Type":"ContainerDied","Data":"9622e41db8ec6f12dd8f4abf928f2ac5f5aaf36001a16f944b9fb7c9a9914aa8"}
Sep 11 12:12:46.131566 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:46.131542    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k" event={"ID":"4db8d0ed-5ed2-4e2f-b3af-ba12172819ce","Type":"ContainerDied","Data":"f85b8dcc70b76b96df2024b8617d5a10fb1e50236fdbb91be965d927d8c9cb01"}
Sep 11 12:12:46.131566 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:46.131559    2569 scope.go:117] "RemoveContainer" containerID="9622e41db8ec6f12dd8f4abf928f2ac5f5aaf36001a16f944b9fb7c9a9914aa8"
Sep 11 12:12:46.141037 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:46.141014    2569 scope.go:117] "RemoveContainer" containerID="9622e41db8ec6f12dd8f4abf928f2ac5f5aaf36001a16f944b9fb7c9a9914aa8"
Sep 11 12:12:46.141300 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:12:46.141279    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9622e41db8ec6f12dd8f4abf928f2ac5f5aaf36001a16f944b9fb7c9a9914aa8\": container with ID starting with 9622e41db8ec6f12dd8f4abf928f2ac5f5aaf36001a16f944b9fb7c9a9914aa8 not found: ID does not exist" containerID="9622e41db8ec6f12dd8f4abf928f2ac5f5aaf36001a16f944b9fb7c9a9914aa8"
Sep 11 12:12:46.141355 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:46.141308    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9622e41db8ec6f12dd8f4abf928f2ac5f5aaf36001a16f944b9fb7c9a9914aa8"} err="failed to get container status \"9622e41db8ec6f12dd8f4abf928f2ac5f5aaf36001a16f944b9fb7c9a9914aa8\": rpc error: code = NotFound desc = could not find container \"9622e41db8ec6f12dd8f4abf928f2ac5f5aaf36001a16f944b9fb7c9a9914aa8\": container with ID starting with 9622e41db8ec6f12dd8f4abf928f2ac5f5aaf36001a16f944b9fb7c9a9914aa8 not found: ID does not exist"
Sep 11 12:12:46.150149 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:46.150127    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"]
Sep 11 12:12:46.153620 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:46.153597    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/sub-test-model-d844cc-kserve-6b9bd86f87-qhc8k"]
Sep 11 12:12:48.070048 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:48.070016    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4db8d0ed-5ed2-4e2f-b3af-ba12172819ce" path="/var/lib/kubelet/pods/4db8d0ed-5ed2-4e2f-b3af-ba12172819ce/volumes"
Sep 11 12:12:49.970909 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:49.970876    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"]
Sep 11 12:12:49.971394 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:49.971299    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db8d0ed-5ed2-4e2f-b3af-ba12172819ce" containerName="main"
Sep 11 12:12:49.971394 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:49.971310    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db8d0ed-5ed2-4e2f-b3af-ba12172819ce" containerName="main"
Sep 11 12:12:49.971394 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:49.971370    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db8d0ed-5ed2-4e2f-b3af-ba12172819ce" containerName="main"
Sep 11 12:12:49.975928 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:49.975909    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:49.979391 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:49.979366    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-af081c\"/\"kube-root-ca.crt\""
Sep 11 12:12:49.979536 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:49.979366    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-af081c\"/\"openshift-service-ca.crt\""
Sep 11 12:12:49.979536 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:49.979440    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-af081c\"/\"rate-test-model-af081c-kserve-self-signed-certs\""
Sep 11 12:12:49.979536 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:49.979401    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-af081c\"/\"default-dockercfg-qbg9d\""
Sep 11 12:12:49.985900 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:49.985881    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"]
Sep 11 12:12:50.054983 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.054958    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a1f8cf-721d-42e7-8355-4fe798963276-tls-certs\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.055131 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.055008    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-tmp-dir\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.055131 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.055063    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-model-cache\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.055224 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.055160    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-dshm\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.055224 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.055187    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-home\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.055323 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.055227    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g9x5\" (UniqueName: \"kubernetes.io/projected/a7a1f8cf-721d-42e7-8355-4fe798963276-kube-api-access-2g9x5\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.156179 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.156157    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a1f8cf-721d-42e7-8355-4fe798963276-tls-certs\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.156324 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.156219    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-tmp-dir\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.156398 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.156355    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-model-cache\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.156462 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.156419    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-dshm\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.156542 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.156519    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-home\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.156620 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.156559    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-tmp-dir\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.156683 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.156659    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g9x5\" (UniqueName: \"kubernetes.io/projected/a7a1f8cf-721d-42e7-8355-4fe798963276-kube-api-access-2g9x5\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.156740 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.156705    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-model-cache\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.156868 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.156849    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-home\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.158543 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.158515    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-dshm\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.158833 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.158816    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a1f8cf-721d-42e7-8355-4fe798963276-tls-certs\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.169183 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.169160    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g9x5\" (UniqueName: \"kubernetes.io/projected/a7a1f8cf-721d-42e7-8355-4fe798963276-kube-api-access-2g9x5\") pod \"rate-test-model-af081c-kserve-745578c97f-8ck2s\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") " pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.285746 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.285722    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:12:50.569535 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:50.563452    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"]
Sep 11 12:12:51.163483 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:51.163446    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s" event={"ID":"a7a1f8cf-721d-42e7-8355-4fe798963276","Type":"ContainerStarted","Data":"3ad0e3fa376c1968eb191003d73cac2513318dd31b559a7abb0306098b521ac3"}
Sep 11 12:12:51.163874 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:51.163498    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s" event={"ID":"a7a1f8cf-721d-42e7-8355-4fe798963276","Type":"ContainerStarted","Data":"f38b8f5d751f4d67c51a59d4264e90580af63b4f614729463cca24a44eb61873"}
Sep 11 12:12:51.185233 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:51.185190    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s" podStartSLOduration=2.185175076 podStartE2EDuration="2.185175076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:12:51.183086665 +0000 UTC m=+1493.604955055" watchObservedRunningTime="2026-09-11 12:12:51.185175076 +0000 UTC m=+1493.607043496"
Sep 11 12:12:58.183465 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:58.183435    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 12:12:58.188367 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:58.188344    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 12:12:58.393125 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:58.393091    2569 scope.go:117] "RemoveContainer" containerID="ffe61f3074e5bb2e2a980aa16b70d3620cc023c68134b86cd2a9a985dca8a460"
Sep 11 12:12:58.404010 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:12:58.403960    2569 scope.go:117] "RemoveContainer" containerID="20fe6dc28223edb8000b918c94aef0a35885e547bc40b28b42ae6a7f03ee705a"
Sep 11 12:13:00.286558 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.286529    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:13:00.286558 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.286567    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:13:00.300200 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.300176    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:13:00.535381 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.535322    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"]
Sep 11 12:13:00.539213 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.539165    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:13:00.542341 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.542321    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f4d189-serving-cert\""
Sep 11 12:13:00.553289 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.553260    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"]
Sep 11 12:13:00.658645 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.658607    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac7367a-e802-4540-877f-cf9048f8ef9b-metrics-certs\") pod \"maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q\" (UID: \"0ac7367a-e802-4540-877f-cf9048f8ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:13:00.658817 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.658657    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac7367a-e802-4540-877f-cf9048f8ef9b-maas-api-tls\") pod \"maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q\" (UID: \"0ac7367a-e802-4540-877f-cf9048f8ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:13:00.658817 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.658768    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28r4b\" (UniqueName: \"kubernetes.io/projected/0ac7367a-e802-4540-877f-cf9048f8ef9b-kube-api-access-28r4b\") pod \"maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q\" (UID: \"0ac7367a-e802-4540-877f-cf9048f8ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:13:00.658817 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.658795    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac7367a-e802-4540-877f-cf9048f8ef9b-openshift-service-ca\") pod \"maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q\" (UID: \"0ac7367a-e802-4540-877f-cf9048f8ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:13:00.759774 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.759735    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28r4b\" (UniqueName: \"kubernetes.io/projected/0ac7367a-e802-4540-877f-cf9048f8ef9b-kube-api-access-28r4b\") pod \"maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q\" (UID: \"0ac7367a-e802-4540-877f-cf9048f8ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:13:00.759940 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.759789    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac7367a-e802-4540-877f-cf9048f8ef9b-openshift-service-ca\") pod \"maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q\" (UID: \"0ac7367a-e802-4540-877f-cf9048f8ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:13:00.759940 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.759871    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac7367a-e802-4540-877f-cf9048f8ef9b-metrics-certs\") pod \"maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q\" (UID: \"0ac7367a-e802-4540-877f-cf9048f8ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:13:00.759940 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.759906    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac7367a-e802-4540-877f-cf9048f8ef9b-maas-api-tls\") pod \"maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q\" (UID: \"0ac7367a-e802-4540-877f-cf9048f8ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:13:00.760066 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:13:00.760056    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-serving-cert: secret "maas-api-e2e-api-b-f4d189-serving-cert" not found
Sep 11 12:13:00.760141 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:13:00.760130    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ac7367a-e802-4540-877f-cf9048f8ef9b-maas-api-tls podName:0ac7367a-e802-4540-877f-cf9048f8ef9b nodeName:}" failed. No retries permitted until 2026-09-11 12:13:01.2601094 +0000 UTC m=+1503.681977766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ac7367a-e802-4540-877f-cf9048f8ef9b-maas-api-tls") pod "maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q" (UID: "0ac7367a-e802-4540-877f-cf9048f8ef9b") : secret "maas-api-e2e-api-b-f4d189-serving-cert" not found
Sep 11 12:13:00.760462 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.760435    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac7367a-e802-4540-877f-cf9048f8ef9b-openshift-service-ca\") pod \"maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q\" (UID: \"0ac7367a-e802-4540-877f-cf9048f8ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:13:00.762375 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.762355    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac7367a-e802-4540-877f-cf9048f8ef9b-metrics-certs\") pod \"maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q\" (UID: \"0ac7367a-e802-4540-877f-cf9048f8ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:13:00.768909 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:00.768876    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28r4b\" (UniqueName: \"kubernetes.io/projected/0ac7367a-e802-4540-877f-cf9048f8ef9b-kube-api-access-28r4b\") pod \"maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q\" (UID: \"0ac7367a-e802-4540-877f-cf9048f8ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:13:01.231093 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:01.231066    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:13:01.264922 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:01.264891    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac7367a-e802-4540-877f-cf9048f8ef9b-maas-api-tls\") pod \"maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q\" (UID: \"0ac7367a-e802-4540-877f-cf9048f8ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:13:01.267836 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:01.267807    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac7367a-e802-4540-877f-cf9048f8ef9b-maas-api-tls\") pod \"maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q\" (UID: \"0ac7367a-e802-4540-877f-cf9048f8ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:13:01.450103 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:01.450074    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:13:01.730429 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:01.730405    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"]
Sep 11 12:13:01.731342 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:13:01.731315    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac7367a_e802_4540_877f_cf9048f8ef9b.slice/crio-44a22650fb1914d1ecdd42e6012493f059cc158bf6c0e3de12353105122ea177 WatchSource:0}: Error finding container 44a22650fb1914d1ecdd42e6012493f059cc158bf6c0e3de12353105122ea177: Status 404 returned error can't find the container with id 44a22650fb1914d1ecdd42e6012493f059cc158bf6c0e3de12353105122ea177
Sep 11 12:13:02.229327 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:02.229291    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q" event={"ID":"0ac7367a-e802-4540-877f-cf9048f8ef9b","Type":"ContainerStarted","Data":"90c73bb810ce6b0ba7297aad3bba3e2ef813aa4f40b031ee52e44f5496ddb0f2"}
Sep 11 12:13:02.229428 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:02.229356    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q" event={"ID":"0ac7367a-e802-4540-877f-cf9048f8ef9b","Type":"ContainerStarted","Data":"44a22650fb1914d1ecdd42e6012493f059cc158bf6c0e3de12353105122ea177"}
Sep 11 12:13:02.229600 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:02.229570    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:13:02.254763 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:02.254699    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q" podStartSLOduration=1.825033177 podStartE2EDuration="2.254679112s" totalImagesPullingTime="429.645935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:13:02.25005812 +0000 UTC m=+1504.671926525" watchObservedRunningTime="2026-09-11 12:13:02.254679112 +0000 UTC m=+1504.676547493"
Sep 11 12:13:08.243306 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:08.243269    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:13:22.758503 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.758427    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"]
Sep 11 12:13:22.762296 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.762281    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.765898 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.765872    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d844cc\"/\"kube-root-ca.crt\""
Sep 11 12:13:22.765898 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.765888    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d844cc\"/\"rate-test-model-d844cc-kserve-self-signed-certs\""
Sep 11 12:13:22.766091 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.765912    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d844cc\"/\"openshift-service-ca.crt\""
Sep 11 12:13:22.766091 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.765871    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d844cc\"/\"default-dockercfg-fl7bt\""
Sep 11 12:13:22.771082 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.771056    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"]
Sep 11 12:13:22.860911 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.860880    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-tmp-dir\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.861055 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.860970    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-model-cache\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.861055 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.861001    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df13740f-d86e-4cbe-b4b6-c9210333bc16-tls-certs\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.861055 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.861033    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwnfz\" (UniqueName: \"kubernetes.io/projected/df13740f-d86e-4cbe-b4b6-c9210333bc16-kube-api-access-gwnfz\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.861198 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.861059    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-home\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.861198 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.861120    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-dshm\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.962511 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.962483    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwnfz\" (UniqueName: \"kubernetes.io/projected/df13740f-d86e-4cbe-b4b6-c9210333bc16-kube-api-access-gwnfz\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.962664 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.962514    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-home\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.962664 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.962546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-dshm\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.962664 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.962626    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-tmp-dir\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.962812 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.962716    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-model-cache\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.962812 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.962744    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df13740f-d86e-4cbe-b4b6-c9210333bc16-tls-certs\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.962985 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.962962    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-home\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.963077 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.963057    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-tmp-dir\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.963128 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.963106    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-model-cache\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.964877 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.964858    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-dshm\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.965233 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.965216    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df13740f-d86e-4cbe-b4b6-c9210333bc16-tls-certs\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:22.971199 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:22.971172    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwnfz\" (UniqueName: \"kubernetes.io/projected/df13740f-d86e-4cbe-b4b6-c9210333bc16-kube-api-access-gwnfz\") pod \"rate-test-model-d844cc-kserve-68d869fc6c-nbh5r\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") " pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:23.072493 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:23.072404    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:23.346286 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:23.346258    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"]
Sep 11 12:13:23.347620 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:13:23.347563    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf13740f_d86e_4cbe_b4b6_c9210333bc16.slice/crio-8d11673f246ee0a1667bc1e0bec297ff840766b88dadb73c12291c6e99ea2c9c WatchSource:0}: Error finding container 8d11673f246ee0a1667bc1e0bec297ff840766b88dadb73c12291c6e99ea2c9c: Status 404 returned error can't find the container with id 8d11673f246ee0a1667bc1e0bec297ff840766b88dadb73c12291c6e99ea2c9c
Sep 11 12:13:24.360044 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:24.360009    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r" event={"ID":"df13740f-d86e-4cbe-b4b6-c9210333bc16","Type":"ContainerStarted","Data":"7a71baf1b0341c7c401e10f05c691e94230032cfc98417f5d1e2b03effa91a25"}
Sep 11 12:13:24.360430 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:24.360051    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r" event={"ID":"df13740f-d86e-4cbe-b4b6-c9210333bc16","Type":"ContainerStarted","Data":"8d11673f246ee0a1667bc1e0bec297ff840766b88dadb73c12291c6e99ea2c9c"}
Sep 11 12:13:24.379831 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:24.379778    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r" podStartSLOduration=2.379762522 podStartE2EDuration="2.379762522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:13:24.378026416 +0000 UTC m=+1526.799894824" watchObservedRunningTime="2026-09-11 12:13:24.379762522 +0000 UTC m=+1526.801630902"
Sep 11 12:13:33.073194 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:33.073144    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:33.073927 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:33.073214    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:33.085475 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:33.085451    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:33.422614 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:33.422530    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:13:42.605104 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:42.605069    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"]
Sep 11 12:13:42.605481 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:42.605280    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl" podUID="cc9dee78-9436-4c0a-918d-91660299eb37" containerName="maas-api" containerID="cri-o://0da718abd7d6c1d38c2c7a20cca857c20234bd3e25e2a25ad7ef979673a9e4f8" gracePeriod=30
Sep 11 12:13:42.859171 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:42.859110    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:13:42.950916 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:42.950873    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc9dee78-9436-4c0a-918d-91660299eb37-openshift-service-ca\") pod \"cc9dee78-9436-4c0a-918d-91660299eb37\" (UID: \"cc9dee78-9436-4c0a-918d-91660299eb37\") "
Sep 11 12:13:42.951097 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:42.950972    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc9dee78-9436-4c0a-918d-91660299eb37-maas-api-tls\") pod \"cc9dee78-9436-4c0a-918d-91660299eb37\" (UID: \"cc9dee78-9436-4c0a-918d-91660299eb37\") "
Sep 11 12:13:42.951097 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:42.951023    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc9dee78-9436-4c0a-918d-91660299eb37-metrics-certs\") pod \"cc9dee78-9436-4c0a-918d-91660299eb37\" (UID: \"cc9dee78-9436-4c0a-918d-91660299eb37\") "
Sep 11 12:13:42.951097 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:42.951075    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hth6\" (UniqueName: \"kubernetes.io/projected/cc9dee78-9436-4c0a-918d-91660299eb37-kube-api-access-8hth6\") pod \"cc9dee78-9436-4c0a-918d-91660299eb37\" (UID: \"cc9dee78-9436-4c0a-918d-91660299eb37\") "
Sep 11 12:13:42.951292 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:42.951209    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9dee78-9436-4c0a-918d-91660299eb37-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc9dee78-9436-4c0a-918d-91660299eb37" (UID: "cc9dee78-9436-4c0a-918d-91660299eb37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:13:42.951450 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:42.951429    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc9dee78-9436-4c0a-918d-91660299eb37-openshift-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:13:42.953193 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:42.953170    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9dee78-9436-4c0a-918d-91660299eb37-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc9dee78-9436-4c0a-918d-91660299eb37" (UID: "cc9dee78-9436-4c0a-918d-91660299eb37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:13:42.953321 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:42.953228    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9dee78-9436-4c0a-918d-91660299eb37-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cc9dee78-9436-4c0a-918d-91660299eb37" (UID: "cc9dee78-9436-4c0a-918d-91660299eb37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:13:42.953321 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:42.953258    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9dee78-9436-4c0a-918d-91660299eb37-kube-api-access-8hth6" (OuterVolumeSpecName: "kube-api-access-8hth6") pod "cc9dee78-9436-4c0a-918d-91660299eb37" (UID: "cc9dee78-9436-4c0a-918d-91660299eb37"). InnerVolumeSpecName "kube-api-access-8hth6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:13:43.052413 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:43.052387    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc9dee78-9436-4c0a-918d-91660299eb37-maas-api-tls\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:13:43.052413 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:43.052410    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc9dee78-9436-4c0a-918d-91660299eb37-metrics-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:13:43.052581 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:43.052421    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hth6\" (UniqueName: \"kubernetes.io/projected/cc9dee78-9436-4c0a-918d-91660299eb37-kube-api-access-8hth6\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:13:43.476697 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:43.476656    2569 generic.go:356] "Generic (PLEG): container finished" podID="cc9dee78-9436-4c0a-918d-91660299eb37" containerID="0da718abd7d6c1d38c2c7a20cca857c20234bd3e25e2a25ad7ef979673a9e4f8" exitCode=0
Sep 11 12:13:43.476901 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:43.476726    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"
Sep 11 12:13:43.476901 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:43.476737    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl" event={"ID":"cc9dee78-9436-4c0a-918d-91660299eb37","Type":"ContainerDied","Data":"0da718abd7d6c1d38c2c7a20cca857c20234bd3e25e2a25ad7ef979673a9e4f8"}
Sep 11 12:13:43.476901 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:43.476817    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl" event={"ID":"cc9dee78-9436-4c0a-918d-91660299eb37","Type":"ContainerDied","Data":"49d1e0be2801a06b3bd1f23f836b80182acf71fa819c9a3c2733fac8ba42f5e1"}
Sep 11 12:13:43.476901 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:43.476839    2569 scope.go:117] "RemoveContainer" containerID="0da718abd7d6c1d38c2c7a20cca857c20234bd3e25e2a25ad7ef979673a9e4f8"
Sep 11 12:13:43.487826 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:43.487806    2569 scope.go:117] "RemoveContainer" containerID="0da718abd7d6c1d38c2c7a20cca857c20234bd3e25e2a25ad7ef979673a9e4f8"
Sep 11 12:13:43.488085 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:13:43.488067    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da718abd7d6c1d38c2c7a20cca857c20234bd3e25e2a25ad7ef979673a9e4f8\": container with ID starting with 0da718abd7d6c1d38c2c7a20cca857c20234bd3e25e2a25ad7ef979673a9e4f8 not found: ID does not exist" containerID="0da718abd7d6c1d38c2c7a20cca857c20234bd3e25e2a25ad7ef979673a9e4f8"
Sep 11 12:13:43.488127 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:43.488093    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da718abd7d6c1d38c2c7a20cca857c20234bd3e25e2a25ad7ef979673a9e4f8"} err="failed to get container status \"0da718abd7d6c1d38c2c7a20cca857c20234bd3e25e2a25ad7ef979673a9e4f8\": rpc error: code = NotFound desc = could not find container \"0da718abd7d6c1d38c2c7a20cca857c20234bd3e25e2a25ad7ef979673a9e4f8\": container with ID starting with 0da718abd7d6c1d38c2c7a20cca857c20234bd3e25e2a25ad7ef979673a9e4f8 not found: ID does not exist"
Sep 11 12:13:43.499912 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:43.499890    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"]
Sep 11 12:13:43.504526 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:43.504506    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fb3c56-59b5ff568-894fl"]
Sep 11 12:13:44.069946 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:13:44.069915    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc9dee78-9436-4c0a-918d-91660299eb37" path="/var/lib/kubelet/pods/cc9dee78-9436-4c0a-918d-91660299eb37/volumes"
Sep 11 12:14:00.991517 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:00.991481    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"]
Sep 11 12:14:01.941565 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:01.941533    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"]
Sep 11 12:14:05.745019 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:05.744987    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"]
Sep 11 12:14:05.745533 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:05.745175    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q" podUID="0ac7367a-e802-4540-877f-cf9048f8ef9b" containerName="maas-api" containerID="cri-o://90c73bb810ce6b0ba7297aad3bba3e2ef813aa4f40b031ee52e44f5496ddb0f2" gracePeriod=30
Sep 11 12:14:05.998964 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:05.998942    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:14:06.070534 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.070501    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac7367a-e802-4540-877f-cf9048f8ef9b-openshift-service-ca\") pod \"0ac7367a-e802-4540-877f-cf9048f8ef9b\" (UID: \"0ac7367a-e802-4540-877f-cf9048f8ef9b\") "
Sep 11 12:14:06.070704 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.070608    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28r4b\" (UniqueName: \"kubernetes.io/projected/0ac7367a-e802-4540-877f-cf9048f8ef9b-kube-api-access-28r4b\") pod \"0ac7367a-e802-4540-877f-cf9048f8ef9b\" (UID: \"0ac7367a-e802-4540-877f-cf9048f8ef9b\") "
Sep 11 12:14:06.070704 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.070641    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac7367a-e802-4540-877f-cf9048f8ef9b-maas-api-tls\") pod \"0ac7367a-e802-4540-877f-cf9048f8ef9b\" (UID: \"0ac7367a-e802-4540-877f-cf9048f8ef9b\") "
Sep 11 12:14:06.070837 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.070708    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac7367a-e802-4540-877f-cf9048f8ef9b-metrics-certs\") pod \"0ac7367a-e802-4540-877f-cf9048f8ef9b\" (UID: \"0ac7367a-e802-4540-877f-cf9048f8ef9b\") "
Sep 11 12:14:06.070936 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.070892    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac7367a-e802-4540-877f-cf9048f8ef9b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0ac7367a-e802-4540-877f-cf9048f8ef9b" (UID: "0ac7367a-e802-4540-877f-cf9048f8ef9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:14:06.071077 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.071059    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac7367a-e802-4540-877f-cf9048f8ef9b-openshift-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:06.072792 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.072758    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac7367a-e802-4540-877f-cf9048f8ef9b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0ac7367a-e802-4540-877f-cf9048f8ef9b" (UID: "0ac7367a-e802-4540-877f-cf9048f8ef9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:14:06.072901 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.072851    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac7367a-e802-4540-877f-cf9048f8ef9b-kube-api-access-28r4b" (OuterVolumeSpecName: "kube-api-access-28r4b") pod "0ac7367a-e802-4540-877f-cf9048f8ef9b" (UID: "0ac7367a-e802-4540-877f-cf9048f8ef9b"). InnerVolumeSpecName "kube-api-access-28r4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:14:06.072901 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.072888    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac7367a-e802-4540-877f-cf9048f8ef9b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0ac7367a-e802-4540-877f-cf9048f8ef9b" (UID: "0ac7367a-e802-4540-877f-cf9048f8ef9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:14:06.172545 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.172517    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28r4b\" (UniqueName: \"kubernetes.io/projected/0ac7367a-e802-4540-877f-cf9048f8ef9b-kube-api-access-28r4b\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:06.172545 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.172540    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac7367a-e802-4540-877f-cf9048f8ef9b-maas-api-tls\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:06.172545 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.172550    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac7367a-e802-4540-877f-cf9048f8ef9b-metrics-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:06.616115 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.616078    2569 generic.go:356] "Generic (PLEG): container finished" podID="0ac7367a-e802-4540-877f-cf9048f8ef9b" containerID="90c73bb810ce6b0ba7297aad3bba3e2ef813aa4f40b031ee52e44f5496ddb0f2" exitCode=0
Sep 11 12:14:06.616290 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.616130    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"
Sep 11 12:14:06.616290 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.616170    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q" event={"ID":"0ac7367a-e802-4540-877f-cf9048f8ef9b","Type":"ContainerDied","Data":"90c73bb810ce6b0ba7297aad3bba3e2ef813aa4f40b031ee52e44f5496ddb0f2"}
Sep 11 12:14:06.616290 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.616226    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q" event={"ID":"0ac7367a-e802-4540-877f-cf9048f8ef9b","Type":"ContainerDied","Data":"44a22650fb1914d1ecdd42e6012493f059cc158bf6c0e3de12353105122ea177"}
Sep 11 12:14:06.616290 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.616248    2569 scope.go:117] "RemoveContainer" containerID="90c73bb810ce6b0ba7297aad3bba3e2ef813aa4f40b031ee52e44f5496ddb0f2"
Sep 11 12:14:06.626652 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.626635    2569 scope.go:117] "RemoveContainer" containerID="90c73bb810ce6b0ba7297aad3bba3e2ef813aa4f40b031ee52e44f5496ddb0f2"
Sep 11 12:14:06.626935 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:14:06.626913    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c73bb810ce6b0ba7297aad3bba3e2ef813aa4f40b031ee52e44f5496ddb0f2\": container with ID starting with 90c73bb810ce6b0ba7297aad3bba3e2ef813aa4f40b031ee52e44f5496ddb0f2 not found: ID does not exist" containerID="90c73bb810ce6b0ba7297aad3bba3e2ef813aa4f40b031ee52e44f5496ddb0f2"
Sep 11 12:14:06.627003 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.626948    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c73bb810ce6b0ba7297aad3bba3e2ef813aa4f40b031ee52e44f5496ddb0f2"} err="failed to get container status \"90c73bb810ce6b0ba7297aad3bba3e2ef813aa4f40b031ee52e44f5496ddb0f2\": rpc error: code = NotFound desc = could not find container \"90c73bb810ce6b0ba7297aad3bba3e2ef813aa4f40b031ee52e44f5496ddb0f2\": container with ID starting with 90c73bb810ce6b0ba7297aad3bba3e2ef813aa4f40b031ee52e44f5496ddb0f2 not found: ID does not exist"
Sep 11 12:14:06.641739 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.641714    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"]
Sep 11 12:14:06.644327 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.644299    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4d189-b5bf4d4b8-r8t8q"]
Sep 11 12:14:06.671633 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.671612    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"]
Sep 11 12:14:06.672696 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.672675    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9dee78-9436-4c0a-918d-91660299eb37" containerName="maas-api"
Sep 11 12:14:06.672696 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.672697    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9dee78-9436-4c0a-918d-91660299eb37" containerName="maas-api"
Sep 11 12:14:06.672869 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.672723    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ac7367a-e802-4540-877f-cf9048f8ef9b" containerName="maas-api"
Sep 11 12:14:06.672869 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.672731    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ac7367a-e802-4540-877f-cf9048f8ef9b" containerName="maas-api"
Sep 11 12:14:06.672869 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.672837    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9dee78-9436-4c0a-918d-91660299eb37" containerName="maas-api"
Sep 11 12:14:06.672869 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.672854    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ac7367a-e802-4540-877f-cf9048f8ef9b" containerName="maas-api"
Sep 11 12:14:06.677431 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.677413    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:14:06.679823 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.679801    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bf9d5c-serving-cert\""
Sep 11 12:14:06.688119 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.688100    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"]
Sep 11 12:14:06.777541 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.777507    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27dxp\" (UniqueName: \"kubernetes.io/projected/b7780da3-ec88-4e47-b90f-4bfc63ff060f-kube-api-access-27dxp\") pod \"maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98\" (UID: \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:14:06.777884 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.777560    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7780da3-ec88-4e47-b90f-4bfc63ff060f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98\" (UID: \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:14:06.777884 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.777671    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7780da3-ec88-4e47-b90f-4bfc63ff060f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98\" (UID: \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:14:06.777884 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.777745    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7780da3-ec88-4e47-b90f-4bfc63ff060f-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98\" (UID: \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:14:06.878289 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.878211    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27dxp\" (UniqueName: \"kubernetes.io/projected/b7780da3-ec88-4e47-b90f-4bfc63ff060f-kube-api-access-27dxp\") pod \"maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98\" (UID: \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:14:06.878289 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.878251    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7780da3-ec88-4e47-b90f-4bfc63ff060f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98\" (UID: \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:14:06.878489 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.878363    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7780da3-ec88-4e47-b90f-4bfc63ff060f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98\" (UID: \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:14:06.878554 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.878517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7780da3-ec88-4e47-b90f-4bfc63ff060f-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98\" (UID: \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:14:06.878923 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.878902    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7780da3-ec88-4e47-b90f-4bfc63ff060f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98\" (UID: \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:14:06.880836 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.880816    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7780da3-ec88-4e47-b90f-4bfc63ff060f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98\" (UID: \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:14:06.880993 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.880976    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7780da3-ec88-4e47-b90f-4bfc63ff060f-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98\" (UID: \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:14:06.886679 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.886658    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27dxp\" (UniqueName: \"kubernetes.io/projected/b7780da3-ec88-4e47-b90f-4bfc63ff060f-kube-api-access-27dxp\") pod \"maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98\" (UID: \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:14:06.990380 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:06.990360    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:14:07.269430 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:07.269395    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"]
Sep 11 12:14:07.270703 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:14:07.270656    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7780da3_ec88_4e47_b90f_4bfc63ff060f.slice/crio-ca381c8f2ac6a2cc63f2b4581fe3ea95ea9a9a8b6abadd58189b25f2e3085442 WatchSource:0}: Error finding container ca381c8f2ac6a2cc63f2b4581fe3ea95ea9a9a8b6abadd58189b25f2e3085442: Status 404 returned error can't find the container with id ca381c8f2ac6a2cc63f2b4581fe3ea95ea9a9a8b6abadd58189b25f2e3085442
Sep 11 12:14:07.625105 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:07.625013    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98" event={"ID":"b7780da3-ec88-4e47-b90f-4bfc63ff060f","Type":"ContainerStarted","Data":"ca381c8f2ac6a2cc63f2b4581fe3ea95ea9a9a8b6abadd58189b25f2e3085442"}
Sep 11 12:14:08.070858 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:08.070827    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ac7367a-e802-4540-877f-cf9048f8ef9b" path="/var/lib/kubelet/pods/0ac7367a-e802-4540-877f-cf9048f8ef9b/volumes"
Sep 11 12:14:08.632549 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:08.632513    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98" event={"ID":"b7780da3-ec88-4e47-b90f-4bfc63ff060f","Type":"ContainerStarted","Data":"00ed6b701d819b187806ec2abf6d502b6ef4c5f70b7f1f2fcb13065878260070"}
Sep 11 12:14:08.632752 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:08.632579    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:14:08.653117 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:08.653067    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98" podStartSLOduration=2.268696637 podStartE2EDuration="2.653051384s" totalImagesPullingTime="384.354747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:08.651038284 +0000 UTC m=+1571.072906676" watchObservedRunningTime="2026-09-11 12:14:08.653051384 +0000 UTC m=+1571.074919816"
Sep 11 12:14:14.644799 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:14.644766    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:14:16.011051 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.010984    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s" podUID="a7a1f8cf-721d-42e7-8355-4fe798963276" containerName="main" containerID="cri-o://3ad0e3fa376c1968eb191003d73cac2513318dd31b559a7abb0306098b521ac3" gracePeriod=45
Sep 11 12:14:16.259216 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.259185    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:14:16.368874 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.368792    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-model-cache\") pod \"a7a1f8cf-721d-42e7-8355-4fe798963276\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") "
Sep 11 12:14:16.368874 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.368852    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-dshm\") pod \"a7a1f8cf-721d-42e7-8355-4fe798963276\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") "
Sep 11 12:14:16.369066 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.368898    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g9x5\" (UniqueName: \"kubernetes.io/projected/a7a1f8cf-721d-42e7-8355-4fe798963276-kube-api-access-2g9x5\") pod \"a7a1f8cf-721d-42e7-8355-4fe798963276\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") "
Sep 11 12:14:16.369066 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.369053    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-tmp-dir\") pod \"a7a1f8cf-721d-42e7-8355-4fe798963276\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") "
Sep 11 12:14:16.369145 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.369096    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7a1f8cf-721d-42e7-8355-4fe798963276" (UID: "a7a1f8cf-721d-42e7-8355-4fe798963276"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:14:16.369145 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.369126    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a1f8cf-721d-42e7-8355-4fe798963276-tls-certs\") pod \"a7a1f8cf-721d-42e7-8355-4fe798963276\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") "
Sep 11 12:14:16.369255 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.369168    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-home\") pod \"a7a1f8cf-721d-42e7-8355-4fe798963276\" (UID: \"a7a1f8cf-721d-42e7-8355-4fe798963276\") "
Sep 11 12:14:16.369355 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.369331    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7a1f8cf-721d-42e7-8355-4fe798963276" (UID: "a7a1f8cf-721d-42e7-8355-4fe798963276"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:14:16.369474 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.369440    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-home" (OuterVolumeSpecName: "home") pod "a7a1f8cf-721d-42e7-8355-4fe798963276" (UID: "a7a1f8cf-721d-42e7-8355-4fe798963276"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:14:16.369571 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.369550    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-tmp-dir\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:16.369571 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.369571    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-home\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:16.369770 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.369607    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-model-cache\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:16.371132 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.371110    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-dshm" (OuterVolumeSpecName: "dshm") pod "a7a1f8cf-721d-42e7-8355-4fe798963276" (UID: "a7a1f8cf-721d-42e7-8355-4fe798963276"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:14:16.371215 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.371131    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a1f8cf-721d-42e7-8355-4fe798963276-kube-api-access-2g9x5" (OuterVolumeSpecName: "kube-api-access-2g9x5") pod "a7a1f8cf-721d-42e7-8355-4fe798963276" (UID: "a7a1f8cf-721d-42e7-8355-4fe798963276"). InnerVolumeSpecName "kube-api-access-2g9x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:14:16.371215 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.371164    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a1f8cf-721d-42e7-8355-4fe798963276-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7a1f8cf-721d-42e7-8355-4fe798963276" (UID: "a7a1f8cf-721d-42e7-8355-4fe798963276"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:14:16.470465 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.470440    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a1f8cf-721d-42e7-8355-4fe798963276-dshm\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:16.470465 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.470464    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g9x5\" (UniqueName: \"kubernetes.io/projected/a7a1f8cf-721d-42e7-8355-4fe798963276-kube-api-access-2g9x5\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:16.470641 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.470474    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a1f8cf-721d-42e7-8355-4fe798963276-tls-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:16.687249 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.687153    2569 generic.go:356] "Generic (PLEG): container finished" podID="a7a1f8cf-721d-42e7-8355-4fe798963276" containerID="3ad0e3fa376c1968eb191003d73cac2513318dd31b559a7abb0306098b521ac3" exitCode=0
Sep 11 12:14:16.687249 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.687191    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"
Sep 11 12:14:16.687249 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.687221    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s" event={"ID":"a7a1f8cf-721d-42e7-8355-4fe798963276","Type":"ContainerDied","Data":"3ad0e3fa376c1968eb191003d73cac2513318dd31b559a7abb0306098b521ac3"}
Sep 11 12:14:16.687538 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.687259    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s" event={"ID":"a7a1f8cf-721d-42e7-8355-4fe798963276","Type":"ContainerDied","Data":"f38b8f5d751f4d67c51a59d4264e90580af63b4f614729463cca24a44eb61873"}
Sep 11 12:14:16.687538 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.687287    2569 scope.go:117] "RemoveContainer" containerID="3ad0e3fa376c1968eb191003d73cac2513318dd31b559a7abb0306098b521ac3"
Sep 11 12:14:16.696954 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.696935    2569 scope.go:117] "RemoveContainer" containerID="3ad0e3fa376c1968eb191003d73cac2513318dd31b559a7abb0306098b521ac3"
Sep 11 12:14:16.697224 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:14:16.697205    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad0e3fa376c1968eb191003d73cac2513318dd31b559a7abb0306098b521ac3\": container with ID starting with 3ad0e3fa376c1968eb191003d73cac2513318dd31b559a7abb0306098b521ac3 not found: ID does not exist" containerID="3ad0e3fa376c1968eb191003d73cac2513318dd31b559a7abb0306098b521ac3"
Sep 11 12:14:16.697271 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.697233    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad0e3fa376c1968eb191003d73cac2513318dd31b559a7abb0306098b521ac3"} err="failed to get container status \"3ad0e3fa376c1968eb191003d73cac2513318dd31b559a7abb0306098b521ac3\": rpc error: code = NotFound desc = could not find container \"3ad0e3fa376c1968eb191003d73cac2513318dd31b559a7abb0306098b521ac3\": container with ID starting with 3ad0e3fa376c1968eb191003d73cac2513318dd31b559a7abb0306098b521ac3 not found: ID does not exist"
Sep 11 12:14:16.709934 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.709909    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"]
Sep 11 12:14:16.716277 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.716256    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af081c/rate-test-model-af081c-kserve-745578c97f-8ck2s"]
Sep 11 12:14:16.961106 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:16.961004    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r" podUID="df13740f-d86e-4cbe-b4b6-c9210333bc16" containerName="main" containerID="cri-o://7a71baf1b0341c7c401e10f05c691e94230032cfc98417f5d1e2b03effa91a25" gracePeriod=45
Sep 11 12:14:17.212739 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.212680    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:14:17.380282 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.380255    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-tmp-dir\") pod \"df13740f-d86e-4cbe-b4b6-c9210333bc16\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") "
Sep 11 12:14:17.380472 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.380301    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-dshm\") pod \"df13740f-d86e-4cbe-b4b6-c9210333bc16\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") "
Sep 11 12:14:17.380472 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.380338    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df13740f-d86e-4cbe-b4b6-c9210333bc16-tls-certs\") pod \"df13740f-d86e-4cbe-b4b6-c9210333bc16\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") "
Sep 11 12:14:17.380472 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.380369    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-home\") pod \"df13740f-d86e-4cbe-b4b6-c9210333bc16\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") "
Sep 11 12:14:17.380472 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.380398    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-model-cache\") pod \"df13740f-d86e-4cbe-b4b6-c9210333bc16\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") "
Sep 11 12:14:17.380741 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.380530    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwnfz\" (UniqueName: \"kubernetes.io/projected/df13740f-d86e-4cbe-b4b6-c9210333bc16-kube-api-access-gwnfz\") pod \"df13740f-d86e-4cbe-b4b6-c9210333bc16\" (UID: \"df13740f-d86e-4cbe-b4b6-c9210333bc16\") "
Sep 11 12:14:17.380741 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.380552    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df13740f-d86e-4cbe-b4b6-c9210333bc16" (UID: "df13740f-d86e-4cbe-b4b6-c9210333bc16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:14:17.380741 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.380685    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-home" (OuterVolumeSpecName: "home") pod "df13740f-d86e-4cbe-b4b6-c9210333bc16" (UID: "df13740f-d86e-4cbe-b4b6-c9210333bc16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:14:17.380741 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.380696    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-model-cache" (OuterVolumeSpecName: "model-cache") pod "df13740f-d86e-4cbe-b4b6-c9210333bc16" (UID: "df13740f-d86e-4cbe-b4b6-c9210333bc16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:14:17.380952 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.380902    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-tmp-dir\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:17.380952 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.380920    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-home\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:17.380952 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.380931    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-model-cache\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:17.382553 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.382532    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-dshm" (OuterVolumeSpecName: "dshm") pod "df13740f-d86e-4cbe-b4b6-c9210333bc16" (UID: "df13740f-d86e-4cbe-b4b6-c9210333bc16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:14:17.382553 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.382530    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df13740f-d86e-4cbe-b4b6-c9210333bc16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df13740f-d86e-4cbe-b4b6-c9210333bc16" (UID: "df13740f-d86e-4cbe-b4b6-c9210333bc16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:14:17.382719 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.382601    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df13740f-d86e-4cbe-b4b6-c9210333bc16-kube-api-access-gwnfz" (OuterVolumeSpecName: "kube-api-access-gwnfz") pod "df13740f-d86e-4cbe-b4b6-c9210333bc16" (UID: "df13740f-d86e-4cbe-b4b6-c9210333bc16"). InnerVolumeSpecName "kube-api-access-gwnfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:14:17.482092 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.482037    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df13740f-d86e-4cbe-b4b6-c9210333bc16-dshm\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:17.482092 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.482058    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df13740f-d86e-4cbe-b4b6-c9210333bc16-tls-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:17.482092 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.482068    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwnfz\" (UniqueName: \"kubernetes.io/projected/df13740f-d86e-4cbe-b4b6-c9210333bc16-kube-api-access-gwnfz\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:17.694597 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.694549    2569 generic.go:356] "Generic (PLEG): container finished" podID="df13740f-d86e-4cbe-b4b6-c9210333bc16" containerID="7a71baf1b0341c7c401e10f05c691e94230032cfc98417f5d1e2b03effa91a25" exitCode=0
Sep 11 12:14:17.694792 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.694631    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"
Sep 11 12:14:17.694792 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.694626    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r" event={"ID":"df13740f-d86e-4cbe-b4b6-c9210333bc16","Type":"ContainerDied","Data":"7a71baf1b0341c7c401e10f05c691e94230032cfc98417f5d1e2b03effa91a25"}
Sep 11 12:14:17.694792 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.694765    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r" event={"ID":"df13740f-d86e-4cbe-b4b6-c9210333bc16","Type":"ContainerDied","Data":"8d11673f246ee0a1667bc1e0bec297ff840766b88dadb73c12291c6e99ea2c9c"}
Sep 11 12:14:17.695020 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.694794    2569 scope.go:117] "RemoveContainer" containerID="7a71baf1b0341c7c401e10f05c691e94230032cfc98417f5d1e2b03effa91a25"
Sep 11 12:14:17.704553 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.704537    2569 scope.go:117] "RemoveContainer" containerID="7a71baf1b0341c7c401e10f05c691e94230032cfc98417f5d1e2b03effa91a25"
Sep 11 12:14:17.704810 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:14:17.704789    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a71baf1b0341c7c401e10f05c691e94230032cfc98417f5d1e2b03effa91a25\": container with ID starting with 7a71baf1b0341c7c401e10f05c691e94230032cfc98417f5d1e2b03effa91a25 not found: ID does not exist" containerID="7a71baf1b0341c7c401e10f05c691e94230032cfc98417f5d1e2b03effa91a25"
Sep 11 12:14:17.704874 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.704816    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a71baf1b0341c7c401e10f05c691e94230032cfc98417f5d1e2b03effa91a25"} err="failed to get container status \"7a71baf1b0341c7c401e10f05c691e94230032cfc98417f5d1e2b03effa91a25\": rpc error: code = NotFound desc = could not find container \"7a71baf1b0341c7c401e10f05c691e94230032cfc98417f5d1e2b03effa91a25\": container with ID starting with 7a71baf1b0341c7c401e10f05c691e94230032cfc98417f5d1e2b03effa91a25 not found: ID does not exist"
Sep 11 12:14:17.716861 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.716838    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"]
Sep 11 12:14:17.720177 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:17.720154    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d844cc/rate-test-model-d844cc-kserve-68d869fc6c-nbh5r"]
Sep 11 12:14:18.070084 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:18.070049    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7a1f8cf-721d-42e7-8355-4fe798963276" path="/var/lib/kubelet/pods/a7a1f8cf-721d-42e7-8355-4fe798963276/volumes"
Sep 11 12:14:18.070442 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:18.070428    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df13740f-d86e-4cbe-b4b6-c9210333bc16" path="/var/lib/kubelet/pods/df13740f-d86e-4cbe-b4b6-c9210333bc16/volumes"
Sep 11 12:14:23.480869 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.480838    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"]
Sep 11 12:14:23.481307 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.481173    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh" podUID="7fe9c22f-b313-419e-9eb0-dcac0553ab2c" containerName="maas-api" containerID="cri-o://cc65d5d415e22a5300c899bfc9f56d7f94c764252c2841b15234bffe7cbb994f" gracePeriod=30
Sep 11 12:14:23.730654 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.730629    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:14:23.732404 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.732332    2569 generic.go:356] "Generic (PLEG): container finished" podID="7fe9c22f-b313-419e-9eb0-dcac0553ab2c" containerID="cc65d5d415e22a5300c899bfc9f56d7f94c764252c2841b15234bffe7cbb994f" exitCode=0
Sep 11 12:14:23.732404 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.732368    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"
Sep 11 12:14:23.732544 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.732417    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh" event={"ID":"7fe9c22f-b313-419e-9eb0-dcac0553ab2c","Type":"ContainerDied","Data":"cc65d5d415e22a5300c899bfc9f56d7f94c764252c2841b15234bffe7cbb994f"}
Sep 11 12:14:23.732544 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.732476    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh" event={"ID":"7fe9c22f-b313-419e-9eb0-dcac0553ab2c","Type":"ContainerDied","Data":"437d89b09904442e1bccb6a9e0f1d215919586f1f4ea3b3590a512e49b3abd47"}
Sep 11 12:14:23.732544 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.732500    2569 scope.go:117] "RemoveContainer" containerID="cc65d5d415e22a5300c899bfc9f56d7f94c764252c2841b15234bffe7cbb994f"
Sep 11 12:14:23.742445 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.742423    2569 scope.go:117] "RemoveContainer" containerID="cc65d5d415e22a5300c899bfc9f56d7f94c764252c2841b15234bffe7cbb994f"
Sep 11 12:14:23.742752 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:14:23.742719    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc65d5d415e22a5300c899bfc9f56d7f94c764252c2841b15234bffe7cbb994f\": container with ID starting with cc65d5d415e22a5300c899bfc9f56d7f94c764252c2841b15234bffe7cbb994f not found: ID does not exist" containerID="cc65d5d415e22a5300c899bfc9f56d7f94c764252c2841b15234bffe7cbb994f"
Sep 11 12:14:23.742855 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.742760    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc65d5d415e22a5300c899bfc9f56d7f94c764252c2841b15234bffe7cbb994f"} err="failed to get container status \"cc65d5d415e22a5300c899bfc9f56d7f94c764252c2841b15234bffe7cbb994f\": rpc error: code = NotFound desc = could not find container \"cc65d5d415e22a5300c899bfc9f56d7f94c764252c2841b15234bffe7cbb994f\": container with ID starting with cc65d5d415e22a5300c899bfc9f56d7f94c764252c2841b15234bffe7cbb994f not found: ID does not exist"
Sep 11 12:14:23.843445 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.843416    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-openshift-service-ca\") pod \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\" (UID: \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\") "
Sep 11 12:14:23.843636 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.843467    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-maas-api-tls\") pod \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\" (UID: \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\") "
Sep 11 12:14:23.843636 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.843600    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-metrics-certs\") pod \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\" (UID: \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\") "
Sep 11 12:14:23.843777 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.843705    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bshz\" (UniqueName: \"kubernetes.io/projected/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-kube-api-access-2bshz\") pod \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\" (UID: \"7fe9c22f-b313-419e-9eb0-dcac0553ab2c\") "
Sep 11 12:14:23.843860 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.843832    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7fe9c22f-b313-419e-9eb0-dcac0553ab2c" (UID: "7fe9c22f-b313-419e-9eb0-dcac0553ab2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:14:23.844050 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.844027    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-openshift-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:23.845662 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.845563    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-kube-api-access-2bshz" (OuterVolumeSpecName: "kube-api-access-2bshz") pod "7fe9c22f-b313-419e-9eb0-dcac0553ab2c" (UID: "7fe9c22f-b313-419e-9eb0-dcac0553ab2c"). InnerVolumeSpecName "kube-api-access-2bshz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:14:23.846032 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.846012    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7fe9c22f-b313-419e-9eb0-dcac0553ab2c" (UID: "7fe9c22f-b313-419e-9eb0-dcac0553ab2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:14:23.846094 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.846029    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7fe9c22f-b313-419e-9eb0-dcac0553ab2c" (UID: "7fe9c22f-b313-419e-9eb0-dcac0553ab2c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:14:23.944718 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.944689    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-metrics-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:23.944718 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.944714    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bshz\" (UniqueName: \"kubernetes.io/projected/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-kube-api-access-2bshz\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:23.944878 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:23.944725    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe9c22f-b313-419e-9eb0-dcac0553ab2c-maas-api-tls\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:24.055877 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:24.055846    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"]
Sep 11 12:14:24.060575 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:24.060554    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45352b-7785dcff4c-dr8gh"]
Sep 11 12:14:24.069692 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:24.069661    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fe9c22f-b313-419e-9eb0-dcac0553ab2c" path="/var/lib/kubelet/pods/7fe9c22f-b313-419e-9eb0-dcac0553ab2c/volumes"
Sep 11 12:14:36.072914 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.072876    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"]
Sep 11 12:14:36.073408 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.073280    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a1f8cf-721d-42e7-8355-4fe798963276" containerName="main"
Sep 11 12:14:36.073408 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.073293    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a1f8cf-721d-42e7-8355-4fe798963276" containerName="main"
Sep 11 12:14:36.073408 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.073301    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df13740f-d86e-4cbe-b4b6-c9210333bc16" containerName="main"
Sep 11 12:14:36.073408 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.073306    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="df13740f-d86e-4cbe-b4b6-c9210333bc16" containerName="main"
Sep 11 12:14:36.073408 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.073319    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe9c22f-b313-419e-9eb0-dcac0553ab2c" containerName="maas-api"
Sep 11 12:14:36.073408 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.073325    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe9c22f-b313-419e-9eb0-dcac0553ab2c" containerName="maas-api"
Sep 11 12:14:36.073408 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.073400    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe9c22f-b313-419e-9eb0-dcac0553ab2c" containerName="maas-api"
Sep 11 12:14:36.073408 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.073411    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a1f8cf-721d-42e7-8355-4fe798963276" containerName="main"
Sep 11 12:14:36.073860 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.073417    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="df13740f-d86e-4cbe-b4b6-c9210333bc16" containerName="main"
Sep 11 12:14:36.076509 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.076489    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:14:36.078821 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.078800    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-986cf2-serving-cert\""
Sep 11 12:14:36.089545 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.089523    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"]
Sep 11 12:14:36.148393 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.148358    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc\" (UID: \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:14:36.148534 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.148421    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc\" (UID: \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:14:36.148534 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.148513    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-metrics-certs\") pod \"maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc\" (UID: \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:14:36.148654 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.148546    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlgmh\" (UniqueName: \"kubernetes.io/projected/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-kube-api-access-vlgmh\") pod \"maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc\" (UID: \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:14:36.249421 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.249390    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc\" (UID: \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:14:36.249615 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.249442    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-metrics-certs\") pod \"maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc\" (UID: \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:14:36.249615 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.249461    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlgmh\" (UniqueName: \"kubernetes.io/projected/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-kube-api-access-vlgmh\") pod \"maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc\" (UID: \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:14:36.249615 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.249519    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc\" (UID: \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:14:36.250124 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.250098    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc\" (UID: \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:14:36.252028 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.252004    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc\" (UID: \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:14:36.252111 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.252069    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-metrics-certs\") pod \"maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc\" (UID: \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:14:36.258325 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.258305    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlgmh\" (UniqueName: \"kubernetes.io/projected/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-kube-api-access-vlgmh\") pod \"maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc\" (UID: \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:14:36.391658 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.391546    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:14:36.671203 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.671178    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"]
Sep 11 12:14:36.672096 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:14:36.672064    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c84368_cf8d_4dc1_918b_a79f3a9556cc.slice/crio-4a41d6ba35cfc07945596f6f997130d52abee04caf8311fb8273b22b9c6357a2 WatchSource:0}: Error finding container 4a41d6ba35cfc07945596f6f997130d52abee04caf8311fb8273b22b9c6357a2: Status 404 returned error can't find the container with id 4a41d6ba35cfc07945596f6f997130d52abee04caf8311fb8273b22b9c6357a2
Sep 11 12:14:36.810726 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:36.810686    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc" event={"ID":"b4c84368-cf8d-4dc1-918b-a79f3a9556cc","Type":"ContainerStarted","Data":"4a41d6ba35cfc07945596f6f997130d52abee04caf8311fb8273b22b9c6357a2"}
Sep 11 12:14:37.819088 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:37.819054    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc" event={"ID":"b4c84368-cf8d-4dc1-918b-a79f3a9556cc","Type":"ContainerStarted","Data":"2524ea2ced4069baf9388a8864cc2dc7793daa05f833bd78353cdf96aa459e91"}
Sep 11 12:14:37.819599 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:37.819185    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:14:37.841922 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:37.841838    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc" podStartSLOduration=0.925931117 podStartE2EDuration="1.841821943s" totalImagesPullingTime="915.890826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:37.839742075 +0000 UTC m=+1600.261610474" watchObservedRunningTime="2026-09-11 12:14:37.841821943 +0000 UTC m=+1600.263690323"
Sep 11 12:14:42.228390 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.228356    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"]
Sep 11 12:14:42.228775 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.228631    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq" podUID="05472180-97a7-40f5-9801-c39de1b24470" containerName="maas-api" containerID="cri-o://466c560c29347db9529c90065d80d9a5276f63792ef96c143e9f7a20d28d87d1" gracePeriod=30
Sep 11 12:14:42.475694 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.475671    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:14:42.503410 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.503384    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqndd\" (UniqueName: \"kubernetes.io/projected/05472180-97a7-40f5-9801-c39de1b24470-kube-api-access-nqndd\") pod \"05472180-97a7-40f5-9801-c39de1b24470\" (UID: \"05472180-97a7-40f5-9801-c39de1b24470\") "
Sep 11 12:14:42.503579 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.503445    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05472180-97a7-40f5-9801-c39de1b24470-openshift-service-ca\") pod \"05472180-97a7-40f5-9801-c39de1b24470\" (UID: \"05472180-97a7-40f5-9801-c39de1b24470\") "
Sep 11 12:14:42.503579 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.503502    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05472180-97a7-40f5-9801-c39de1b24470-maas-api-tls\") pod \"05472180-97a7-40f5-9801-c39de1b24470\" (UID: \"05472180-97a7-40f5-9801-c39de1b24470\") "
Sep 11 12:14:42.503579 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.503538    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05472180-97a7-40f5-9801-c39de1b24470-metrics-certs\") pod \"05472180-97a7-40f5-9801-c39de1b24470\" (UID: \"05472180-97a7-40f5-9801-c39de1b24470\") "
Sep 11 12:14:42.503818 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.503791    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05472180-97a7-40f5-9801-c39de1b24470-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "05472180-97a7-40f5-9801-c39de1b24470" (UID: "05472180-97a7-40f5-9801-c39de1b24470"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:14:42.503904 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.503888    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05472180-97a7-40f5-9801-c39de1b24470-openshift-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:42.505502 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.505474    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05472180-97a7-40f5-9801-c39de1b24470-kube-api-access-nqndd" (OuterVolumeSpecName: "kube-api-access-nqndd") pod "05472180-97a7-40f5-9801-c39de1b24470" (UID: "05472180-97a7-40f5-9801-c39de1b24470"). InnerVolumeSpecName "kube-api-access-nqndd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:14:42.505657 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.505631    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05472180-97a7-40f5-9801-c39de1b24470-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "05472180-97a7-40f5-9801-c39de1b24470" (UID: "05472180-97a7-40f5-9801-c39de1b24470"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:14:42.505714 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.505694    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05472180-97a7-40f5-9801-c39de1b24470-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "05472180-97a7-40f5-9801-c39de1b24470" (UID: "05472180-97a7-40f5-9801-c39de1b24470"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:14:42.604745 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.604722    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05472180-97a7-40f5-9801-c39de1b24470-maas-api-tls\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:42.604849 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.604747    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05472180-97a7-40f5-9801-c39de1b24470-metrics-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:42.604849 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.604760    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqndd\" (UniqueName: \"kubernetes.io/projected/05472180-97a7-40f5-9801-c39de1b24470-kube-api-access-nqndd\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:14:42.848803 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.848714    2569 generic.go:356] "Generic (PLEG): container finished" podID="05472180-97a7-40f5-9801-c39de1b24470" containerID="466c560c29347db9529c90065d80d9a5276f63792ef96c143e9f7a20d28d87d1" exitCode=0
Sep 11 12:14:42.848803 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.848765    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"
Sep 11 12:14:42.849000 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.848800    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq" event={"ID":"05472180-97a7-40f5-9801-c39de1b24470","Type":"ContainerDied","Data":"466c560c29347db9529c90065d80d9a5276f63792ef96c143e9f7a20d28d87d1"}
Sep 11 12:14:42.849000 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.848854    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq" event={"ID":"05472180-97a7-40f5-9801-c39de1b24470","Type":"ContainerDied","Data":"693fddb82a83e58bf2334f0b2547e272572470d990a0ff733c8bb3e92dc77e88"}
Sep 11 12:14:42.849000 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.848876    2569 scope.go:117] "RemoveContainer" containerID="466c560c29347db9529c90065d80d9a5276f63792ef96c143e9f7a20d28d87d1"
Sep 11 12:14:42.859465 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.859449    2569 scope.go:117] "RemoveContainer" containerID="466c560c29347db9529c90065d80d9a5276f63792ef96c143e9f7a20d28d87d1"
Sep 11 12:14:42.859727 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:14:42.859709    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466c560c29347db9529c90065d80d9a5276f63792ef96c143e9f7a20d28d87d1\": container with ID starting with 466c560c29347db9529c90065d80d9a5276f63792ef96c143e9f7a20d28d87d1 not found: ID does not exist" containerID="466c560c29347db9529c90065d80d9a5276f63792ef96c143e9f7a20d28d87d1"
Sep 11 12:14:42.859779 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.859734    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466c560c29347db9529c90065d80d9a5276f63792ef96c143e9f7a20d28d87d1"} err="failed to get container status \"466c560c29347db9529c90065d80d9a5276f63792ef96c143e9f7a20d28d87d1\": rpc error: code = NotFound desc = could not find container \"466c560c29347db9529c90065d80d9a5276f63792ef96c143e9f7a20d28d87d1\": container with ID starting with 466c560c29347db9529c90065d80d9a5276f63792ef96c143e9f7a20d28d87d1 not found: ID does not exist"
Sep 11 12:14:42.871968 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.871948    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"]
Sep 11 12:14:42.881674 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:42.881655    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f7851b-64c8b777b8-t2lcq"]
Sep 11 12:14:43.830842 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:43.830816    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:14:44.069859 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:14:44.069828    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05472180-97a7-40f5-9801-c39de1b24470" path="/var/lib/kubelet/pods/05472180-97a7-40f5-9801-c39de1b24470/volumes"
Sep 11 12:15:18.549554 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.549525    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"]
Sep 11 12:15:18.550191 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.550169    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05472180-97a7-40f5-9801-c39de1b24470" containerName="maas-api"
Sep 11 12:15:18.550279 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.550195    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="05472180-97a7-40f5-9801-c39de1b24470" containerName="maas-api"
Sep 11 12:15:18.550375 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.550354    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="05472180-97a7-40f5-9801-c39de1b24470" containerName="maas-api"
Sep 11 12:15:18.555701 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.555676    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.559051 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.559023    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bf9d5c\"/\"ipp-route-bf9d5c-kserve-self-signed-certs\""
Sep 11 12:15:18.559179 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.559097    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bf9d5c\"/\"kube-root-ca.crt\""
Sep 11 12:15:18.559470 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.559310    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bf9d5c\"/\"openshift-service-ca.crt\""
Sep 11 12:15:18.559470 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.559395    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bf9d5c\"/\"default-dockercfg-nmk9m\""
Sep 11 12:15:18.562201 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.562177    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"]
Sep 11 12:15:18.740701 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.740652    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-dshm\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.740883 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.740727    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b9c7a7-88f2-4498-8452-e6c517f192d9-tls-certs\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.740883 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.740764    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-tmp-dir\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.740883 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.740834    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk874\" (UniqueName: \"kubernetes.io/projected/36b9c7a7-88f2-4498-8452-e6c517f192d9-kube-api-access-hk874\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.740993 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.740882    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-home\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.740993 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.740905    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-model-cache\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.842003 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.841915    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-model-cache\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.842003 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.841987    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-dshm\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.842220 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.842050    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b9c7a7-88f2-4498-8452-e6c517f192d9-tls-certs\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.842220 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.842099    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-tmp-dir\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.842220 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.842146    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk874\" (UniqueName: \"kubernetes.io/projected/36b9c7a7-88f2-4498-8452-e6c517f192d9-kube-api-access-hk874\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.842220 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.842195    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-home\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.842442 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.842291    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-model-cache\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.842512 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.842474    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-tmp-dir\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.842611 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.842569    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-home\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.844129 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.844110    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-dshm\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.844380 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.844362    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b9c7a7-88f2-4498-8452-e6c517f192d9-tls-certs\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.851978 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.851952    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk874\" (UniqueName: \"kubernetes.io/projected/36b9c7a7-88f2-4498-8452-e6c517f192d9-kube-api-access-hk874\") pod \"ipp-route-bf9d5c-kserve-5b75849587-mbnqx\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") " pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:18.868775 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:18.868751    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:19.144803 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:19.144777    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"]
Sep 11 12:15:19.147683 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:15:19.147645    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36b9c7a7_88f2_4498_8452_e6c517f192d9.slice/crio-88015fc02825c283fad2e096d834b1083ad453f1226c10fbc0bb855b3d73e4a8 WatchSource:0}: Error finding container 88015fc02825c283fad2e096d834b1083ad453f1226c10fbc0bb855b3d73e4a8: Status 404 returned error can't find the container with id 88015fc02825c283fad2e096d834b1083ad453f1226c10fbc0bb855b3d73e4a8
Sep 11 12:15:20.071089 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:20.071049    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx" event={"ID":"36b9c7a7-88f2-4498-8452-e6c517f192d9","Type":"ContainerStarted","Data":"b7e66cb9db0cf5112e56f8a0d069569c6292e2df674e663a5c55191308f352ca"}
Sep 11 12:15:20.071487 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:20.071098    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx" event={"ID":"36b9c7a7-88f2-4498-8452-e6c517f192d9","Type":"ContainerStarted","Data":"88015fc02825c283fad2e096d834b1083ad453f1226c10fbc0bb855b3d73e4a8"}
Sep 11 12:15:20.091543 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:20.091493    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx" podStartSLOduration=2.0914779980000002 podStartE2EDuration="2.091477998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:20.089442585 +0000 UTC m=+1642.511310989" watchObservedRunningTime="2026-09-11 12:15:20.091477998 +0000 UTC m=+1642.513346378"
Sep 11 12:15:28.869021 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:28.868990    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:28.869519 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:28.869240    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:28.881940 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:28.881915    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:15:29.129244 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:15:29.129164    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:16:57.357859 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.357826    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"]
Sep 11 12:16:57.358455 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.358026    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98" podUID="b7780da3-ec88-4e47-b90f-4bfc63ff060f" containerName="maas-api" containerID="cri-o://00ed6b701d819b187806ec2abf6d502b6ef4c5f70b7f1f2fcb13065878260070" gracePeriod=30
Sep 11 12:16:57.621115 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.621067    2569 generic.go:356] "Generic (PLEG): container finished" podID="b7780da3-ec88-4e47-b90f-4bfc63ff060f" containerID="00ed6b701d819b187806ec2abf6d502b6ef4c5f70b7f1f2fcb13065878260070" exitCode=0
Sep 11 12:16:57.621249 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.621134    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98" event={"ID":"b7780da3-ec88-4e47-b90f-4bfc63ff060f","Type":"ContainerDied","Data":"00ed6b701d819b187806ec2abf6d502b6ef4c5f70b7f1f2fcb13065878260070"}
Sep 11 12:16:57.621249 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.621227    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98" event={"ID":"b7780da3-ec88-4e47-b90f-4bfc63ff060f","Type":"ContainerDied","Data":"ca381c8f2ac6a2cc63f2b4581fe3ea95ea9a9a8b6abadd58189b25f2e3085442"}
Sep 11 12:16:57.621345 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.621252    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca381c8f2ac6a2cc63f2b4581fe3ea95ea9a9a8b6abadd58189b25f2e3085442"
Sep 11 12:16:57.623552 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.623533    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:16:57.675941 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.675916    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7780da3-ec88-4e47-b90f-4bfc63ff060f-openshift-service-ca\") pod \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\" (UID: \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\") "
Sep 11 12:16:57.676146 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.676020    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7780da3-ec88-4e47-b90f-4bfc63ff060f-metrics-certs\") pod \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\" (UID: \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\") "
Sep 11 12:16:57.676146 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.676088    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27dxp\" (UniqueName: \"kubernetes.io/projected/b7780da3-ec88-4e47-b90f-4bfc63ff060f-kube-api-access-27dxp\") pod \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\" (UID: \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\") "
Sep 11 12:16:57.676335 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.676174    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7780da3-ec88-4e47-b90f-4bfc63ff060f-maas-api-tls\") pod \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\" (UID: \"b7780da3-ec88-4e47-b90f-4bfc63ff060f\") "
Sep 11 12:16:57.676335 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.676286    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7780da3-ec88-4e47-b90f-4bfc63ff060f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b7780da3-ec88-4e47-b90f-4bfc63ff060f" (UID: "b7780da3-ec88-4e47-b90f-4bfc63ff060f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:16:57.676512 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.676490    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7780da3-ec88-4e47-b90f-4bfc63ff060f-openshift-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:16:57.678508 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.678477    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7780da3-ec88-4e47-b90f-4bfc63ff060f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b7780da3-ec88-4e47-b90f-4bfc63ff060f" (UID: "b7780da3-ec88-4e47-b90f-4bfc63ff060f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:16:57.678678 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.678655    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7780da3-ec88-4e47-b90f-4bfc63ff060f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b7780da3-ec88-4e47-b90f-4bfc63ff060f" (UID: "b7780da3-ec88-4e47-b90f-4bfc63ff060f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:16:57.679015 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.678990    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7780da3-ec88-4e47-b90f-4bfc63ff060f-kube-api-access-27dxp" (OuterVolumeSpecName: "kube-api-access-27dxp") pod "b7780da3-ec88-4e47-b90f-4bfc63ff060f" (UID: "b7780da3-ec88-4e47-b90f-4bfc63ff060f"). InnerVolumeSpecName "kube-api-access-27dxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:16:57.777509 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.777477    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7780da3-ec88-4e47-b90f-4bfc63ff060f-metrics-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:16:57.777509 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.777505    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27dxp\" (UniqueName: \"kubernetes.io/projected/b7780da3-ec88-4e47-b90f-4bfc63ff060f-kube-api-access-27dxp\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:16:57.777509 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:57.777515    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7780da3-ec88-4e47-b90f-4bfc63ff060f-maas-api-tls\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:16:58.626370 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:58.626341    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"
Sep 11 12:16:58.643829 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:58.643804    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"]
Sep 11 12:16:58.647910 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:16:58.647890    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9d5c-5b445bdd67-mwv98"]
Sep 11 12:17:00.070401 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:00.070368    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7780da3-ec88-4e47-b90f-4bfc63ff060f" path="/var/lib/kubelet/pods/b7780da3-ec88-4e47-b90f-4bfc63ff060f/volumes"
Sep 11 12:17:02.987546 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:02.987504    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"]
Sep 11 12:17:18.014869 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.014825    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx" podUID="36b9c7a7-88f2-4498-8452-e6c517f192d9" containerName="main" containerID="cri-o://b7e66cb9db0cf5112e56f8a0d069569c6292e2df674e663a5c55191308f352ca" gracePeriod=45
Sep 11 12:17:18.259558 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.259534    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:17:18.363464 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.363382    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-tmp-dir\") pod \"36b9c7a7-88f2-4498-8452-e6c517f192d9\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") "
Sep 11 12:17:18.363464 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.363414    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-model-cache\") pod \"36b9c7a7-88f2-4498-8452-e6c517f192d9\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") "
Sep 11 12:17:18.363464 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.363442    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-home\") pod \"36b9c7a7-88f2-4498-8452-e6c517f192d9\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") "
Sep 11 12:17:18.363464 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.363462    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b9c7a7-88f2-4498-8452-e6c517f192d9-tls-certs\") pod \"36b9c7a7-88f2-4498-8452-e6c517f192d9\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") "
Sep 11 12:17:18.363824 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.363486    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hk874\" (UniqueName: \"kubernetes.io/projected/36b9c7a7-88f2-4498-8452-e6c517f192d9-kube-api-access-hk874\") pod \"36b9c7a7-88f2-4498-8452-e6c517f192d9\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") "
Sep 11 12:17:18.363824 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.363664    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-dshm\") pod \"36b9c7a7-88f2-4498-8452-e6c517f192d9\" (UID: \"36b9c7a7-88f2-4498-8452-e6c517f192d9\") "
Sep 11 12:17:18.363824 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.363768    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-home" (OuterVolumeSpecName: "home") pod "36b9c7a7-88f2-4498-8452-e6c517f192d9" (UID: "36b9c7a7-88f2-4498-8452-e6c517f192d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:17:18.363824 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.363776    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "36b9c7a7-88f2-4498-8452-e6c517f192d9" (UID: "36b9c7a7-88f2-4498-8452-e6c517f192d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:17:18.363824 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.363784    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "36b9c7a7-88f2-4498-8452-e6c517f192d9" (UID: "36b9c7a7-88f2-4498-8452-e6c517f192d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:17:18.364089 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.364067    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-tmp-dir\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:17:18.364132 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.364096    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-model-cache\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:17:18.364132 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.364111    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-home\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:17:18.365793 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.365757    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b9c7a7-88f2-4498-8452-e6c517f192d9-kube-api-access-hk874" (OuterVolumeSpecName: "kube-api-access-hk874") pod "36b9c7a7-88f2-4498-8452-e6c517f192d9" (UID: "36b9c7a7-88f2-4498-8452-e6c517f192d9"). InnerVolumeSpecName "kube-api-access-hk874". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:17:18.366134 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.366110    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-dshm" (OuterVolumeSpecName: "dshm") pod "36b9c7a7-88f2-4498-8452-e6c517f192d9" (UID: "36b9c7a7-88f2-4498-8452-e6c517f192d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:17:18.366213 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.366161    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b9c7a7-88f2-4498-8452-e6c517f192d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36b9c7a7-88f2-4498-8452-e6c517f192d9" (UID: "36b9c7a7-88f2-4498-8452-e6c517f192d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:17:18.465355 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.465320    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b9c7a7-88f2-4498-8452-e6c517f192d9-dshm\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:17:18.465355 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.465347    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b9c7a7-88f2-4498-8452-e6c517f192d9-tls-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:17:18.465355 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.465357    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk874\" (UniqueName: \"kubernetes.io/projected/36b9c7a7-88f2-4498-8452-e6c517f192d9-kube-api-access-hk874\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:17:18.752233 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.752201    2569 generic.go:356] "Generic (PLEG): container finished" podID="36b9c7a7-88f2-4498-8452-e6c517f192d9" containerID="b7e66cb9db0cf5112e56f8a0d069569c6292e2df674e663a5c55191308f352ca" exitCode=0
Sep 11 12:17:18.752450 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.752240    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"
Sep 11 12:17:18.752450 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.752286    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx" event={"ID":"36b9c7a7-88f2-4498-8452-e6c517f192d9","Type":"ContainerDied","Data":"b7e66cb9db0cf5112e56f8a0d069569c6292e2df674e663a5c55191308f352ca"}
Sep 11 12:17:18.752450 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.752329    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx" event={"ID":"36b9c7a7-88f2-4498-8452-e6c517f192d9","Type":"ContainerDied","Data":"88015fc02825c283fad2e096d834b1083ad453f1226c10fbc0bb855b3d73e4a8"}
Sep 11 12:17:18.752450 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.752350    2569 scope.go:117] "RemoveContainer" containerID="b7e66cb9db0cf5112e56f8a0d069569c6292e2df674e663a5c55191308f352ca"
Sep 11 12:17:18.762339 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.762321    2569 scope.go:117] "RemoveContainer" containerID="b7e66cb9db0cf5112e56f8a0d069569c6292e2df674e663a5c55191308f352ca"
Sep 11 12:17:18.762623 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:17:18.762601    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7e66cb9db0cf5112e56f8a0d069569c6292e2df674e663a5c55191308f352ca\": container with ID starting with b7e66cb9db0cf5112e56f8a0d069569c6292e2df674e663a5c55191308f352ca not found: ID does not exist" containerID="b7e66cb9db0cf5112e56f8a0d069569c6292e2df674e663a5c55191308f352ca"
Sep 11 12:17:18.762669 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.762632    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e66cb9db0cf5112e56f8a0d069569c6292e2df674e663a5c55191308f352ca"} err="failed to get container status \"b7e66cb9db0cf5112e56f8a0d069569c6292e2df674e663a5c55191308f352ca\": rpc error: code = NotFound desc = could not find container \"b7e66cb9db0cf5112e56f8a0d069569c6292e2df674e663a5c55191308f352ca\": container with ID starting with b7e66cb9db0cf5112e56f8a0d069569c6292e2df674e663a5c55191308f352ca not found: ID does not exist"
Sep 11 12:17:18.777602 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.777561    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"]
Sep 11 12:17:18.784116 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:18.784095    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-bf9d5c/ipp-route-bf9d5c-kserve-5b75849587-mbnqx"]
Sep 11 12:17:20.070389 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:20.070356    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36b9c7a7-88f2-4498-8452-e6c517f192d9" path="/var/lib/kubelet/pods/36b9c7a7-88f2-4498-8452-e6c517f192d9/volumes"
Sep 11 12:17:58.224841 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:58.224814    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 12:17:58.231601 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:17:58.231573    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 12:18:12.402951 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:12.402917    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"]
Sep 11 12:18:12.403487 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:12.403148    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc" podUID="b4c84368-cf8d-4dc1-918b-a79f3a9556cc" containerName="maas-api" containerID="cri-o://2524ea2ced4069baf9388a8864cc2dc7793daa05f833bd78353cdf96aa459e91" gracePeriod=30
Sep 11 12:18:12.653669 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:12.653596    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:18:12.732641 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:12.732604    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-maas-api-tls\") pod \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\" (UID: \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\") "
Sep 11 12:18:12.732784 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:12.732661    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-openshift-service-ca\") pod \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\" (UID: \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\") "
Sep 11 12:18:12.732784 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:12.732705    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-metrics-certs\") pod \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\" (UID: \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\") "
Sep 11 12:18:12.732878 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:12.732824    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlgmh\" (UniqueName: \"kubernetes.io/projected/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-kube-api-access-vlgmh\") pod \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\" (UID: \"b4c84368-cf8d-4dc1-918b-a79f3a9556cc\") "
Sep 11 12:18:12.733093 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:12.733062    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b4c84368-cf8d-4dc1-918b-a79f3a9556cc" (UID: "b4c84368-cf8d-4dc1-918b-a79f3a9556cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:18:12.733225 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:12.733172    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-openshift-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:18:12.734797 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:12.734775    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b4c84368-cf8d-4dc1-918b-a79f3a9556cc" (UID: "b4c84368-cf8d-4dc1-918b-a79f3a9556cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:18:12.734899 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:12.734816    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b4c84368-cf8d-4dc1-918b-a79f3a9556cc" (UID: "b4c84368-cf8d-4dc1-918b-a79f3a9556cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:18:12.734899 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:12.734881    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-kube-api-access-vlgmh" (OuterVolumeSpecName: "kube-api-access-vlgmh") pod "b4c84368-cf8d-4dc1-918b-a79f3a9556cc" (UID: "b4c84368-cf8d-4dc1-918b-a79f3a9556cc"). InnerVolumeSpecName "kube-api-access-vlgmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:18:12.834247 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:12.834212    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlgmh\" (UniqueName: \"kubernetes.io/projected/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-kube-api-access-vlgmh\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:18:12.834247 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:12.834247    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-maas-api-tls\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:18:12.834247 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:12.834256    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4c84368-cf8d-4dc1-918b-a79f3a9556cc-metrics-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:18:13.070518 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:13.070481    2569 generic.go:356] "Generic (PLEG): container finished" podID="b4c84368-cf8d-4dc1-918b-a79f3a9556cc" containerID="2524ea2ced4069baf9388a8864cc2dc7793daa05f833bd78353cdf96aa459e91" exitCode=0
Sep 11 12:18:13.070693 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:13.070521    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"
Sep 11 12:18:13.070693 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:13.070539    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc" event={"ID":"b4c84368-cf8d-4dc1-918b-a79f3a9556cc","Type":"ContainerDied","Data":"2524ea2ced4069baf9388a8864cc2dc7793daa05f833bd78353cdf96aa459e91"}
Sep 11 12:18:13.070693 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:13.070611    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc" event={"ID":"b4c84368-cf8d-4dc1-918b-a79f3a9556cc","Type":"ContainerDied","Data":"4a41d6ba35cfc07945596f6f997130d52abee04caf8311fb8273b22b9c6357a2"}
Sep 11 12:18:13.070693 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:13.070613    2569 scope.go:117] "RemoveContainer" containerID="2524ea2ced4069baf9388a8864cc2dc7793daa05f833bd78353cdf96aa459e91"
Sep 11 12:18:13.080452 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:13.080434    2569 scope.go:117] "RemoveContainer" containerID="2524ea2ced4069baf9388a8864cc2dc7793daa05f833bd78353cdf96aa459e91"
Sep 11 12:18:13.080727 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:18:13.080709    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2524ea2ced4069baf9388a8864cc2dc7793daa05f833bd78353cdf96aa459e91\": container with ID starting with 2524ea2ced4069baf9388a8864cc2dc7793daa05f833bd78353cdf96aa459e91 not found: ID does not exist" containerID="2524ea2ced4069baf9388a8864cc2dc7793daa05f833bd78353cdf96aa459e91"
Sep 11 12:18:13.080789 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:13.080735    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2524ea2ced4069baf9388a8864cc2dc7793daa05f833bd78353cdf96aa459e91"} err="failed to get container status \"2524ea2ced4069baf9388a8864cc2dc7793daa05f833bd78353cdf96aa459e91\": rpc error: code = NotFound desc = could not find container \"2524ea2ced4069baf9388a8864cc2dc7793daa05f833bd78353cdf96aa459e91\": container with ID starting with 2524ea2ced4069baf9388a8864cc2dc7793daa05f833bd78353cdf96aa459e91 not found: ID does not exist"
Sep 11 12:18:13.093896 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:13.093873    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"]
Sep 11 12:18:13.103428 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:13.100996    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-986cf2-6db5c8475f-ncmjc"]
Sep 11 12:18:14.070515 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:14.070470    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4c84368-cf8d-4dc1-918b-a79f3a9556cc" path="/var/lib/kubelet/pods/b4c84368-cf8d-4dc1-918b-a79f3a9556cc/volumes"
Sep 11 12:18:30.602728 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:30.602691    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"]
Sep 11 12:18:30.603128 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:30.602955    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h" podUID="6093502b-ce5d-42aa-9c45-41fe039002c3" containerName="maas-api" containerID="cri-o://3554bb81f9a1969796c40602a45a21f6c38890de981ae92fb5eb50002d787171" gracePeriod=30
Sep 11 12:18:30.852540 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:30.852513    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:18:30.887503 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:30.887425    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6093502b-ce5d-42aa-9c45-41fe039002c3-openshift-service-ca\") pod \"6093502b-ce5d-42aa-9c45-41fe039002c3\" (UID: \"6093502b-ce5d-42aa-9c45-41fe039002c3\") "
Sep 11 12:18:30.887648 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:30.887515    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6093502b-ce5d-42aa-9c45-41fe039002c3-maas-api-tls\") pod \"6093502b-ce5d-42aa-9c45-41fe039002c3\" (UID: \"6093502b-ce5d-42aa-9c45-41fe039002c3\") "
Sep 11 12:18:30.887648 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:30.887553    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fkt4\" (UniqueName: \"kubernetes.io/projected/6093502b-ce5d-42aa-9c45-41fe039002c3-kube-api-access-5fkt4\") pod \"6093502b-ce5d-42aa-9c45-41fe039002c3\" (UID: \"6093502b-ce5d-42aa-9c45-41fe039002c3\") "
Sep 11 12:18:30.887648 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:30.887618    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6093502b-ce5d-42aa-9c45-41fe039002c3-metrics-certs\") pod \"6093502b-ce5d-42aa-9c45-41fe039002c3\" (UID: \"6093502b-ce5d-42aa-9c45-41fe039002c3\") "
Sep 11 12:18:30.887829 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:30.887780    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6093502b-ce5d-42aa-9c45-41fe039002c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6093502b-ce5d-42aa-9c45-41fe039002c3" (UID: "6093502b-ce5d-42aa-9c45-41fe039002c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:18:30.887988 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:30.887962    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6093502b-ce5d-42aa-9c45-41fe039002c3-openshift-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:18:30.889663 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:30.889635    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6093502b-ce5d-42aa-9c45-41fe039002c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6093502b-ce5d-42aa-9c45-41fe039002c3" (UID: "6093502b-ce5d-42aa-9c45-41fe039002c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:18:30.889775 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:30.889700    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6093502b-ce5d-42aa-9c45-41fe039002c3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6093502b-ce5d-42aa-9c45-41fe039002c3" (UID: "6093502b-ce5d-42aa-9c45-41fe039002c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:18:30.889775 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:30.889763    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6093502b-ce5d-42aa-9c45-41fe039002c3-kube-api-access-5fkt4" (OuterVolumeSpecName: "kube-api-access-5fkt4") pod "6093502b-ce5d-42aa-9c45-41fe039002c3" (UID: "6093502b-ce5d-42aa-9c45-41fe039002c3"). InnerVolumeSpecName "kube-api-access-5fkt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:18:30.989185 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:30.989152    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6093502b-ce5d-42aa-9c45-41fe039002c3-metrics-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:18:30.989185 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:30.989178    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6093502b-ce5d-42aa-9c45-41fe039002c3-maas-api-tls\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:18:30.989185 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:30.989187    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fkt4\" (UniqueName: \"kubernetes.io/projected/6093502b-ce5d-42aa-9c45-41fe039002c3-kube-api-access-5fkt4\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:18:31.170068 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:31.169980    2569 generic.go:356] "Generic (PLEG): container finished" podID="6093502b-ce5d-42aa-9c45-41fe039002c3" containerID="3554bb81f9a1969796c40602a45a21f6c38890de981ae92fb5eb50002d787171" exitCode=0
Sep 11 12:18:31.170068 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:31.170020    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"
Sep 11 12:18:31.170255 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:31.170066    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h" event={"ID":"6093502b-ce5d-42aa-9c45-41fe039002c3","Type":"ContainerDied","Data":"3554bb81f9a1969796c40602a45a21f6c38890de981ae92fb5eb50002d787171"}
Sep 11 12:18:31.170255 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:31.170118    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h" event={"ID":"6093502b-ce5d-42aa-9c45-41fe039002c3","Type":"ContainerDied","Data":"0bc6c8b77fd4eef1d4a86d4a31abe18ddb5cbba669dc97c3aafef8db00e2d792"}
Sep 11 12:18:31.170255 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:31.170138    2569 scope.go:117] "RemoveContainer" containerID="3554bb81f9a1969796c40602a45a21f6c38890de981ae92fb5eb50002d787171"
Sep 11 12:18:31.180220 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:31.180204    2569 scope.go:117] "RemoveContainer" containerID="3554bb81f9a1969796c40602a45a21f6c38890de981ae92fb5eb50002d787171"
Sep 11 12:18:31.180488 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:18:31.180465    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3554bb81f9a1969796c40602a45a21f6c38890de981ae92fb5eb50002d787171\": container with ID starting with 3554bb81f9a1969796c40602a45a21f6c38890de981ae92fb5eb50002d787171 not found: ID does not exist" containerID="3554bb81f9a1969796c40602a45a21f6c38890de981ae92fb5eb50002d787171"
Sep 11 12:18:31.180537 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:31.180496    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3554bb81f9a1969796c40602a45a21f6c38890de981ae92fb5eb50002d787171"} err="failed to get container status \"3554bb81f9a1969796c40602a45a21f6c38890de981ae92fb5eb50002d787171\": rpc error: code = NotFound desc = could not find container \"3554bb81f9a1969796c40602a45a21f6c38890de981ae92fb5eb50002d787171\": container with ID starting with 3554bb81f9a1969796c40602a45a21f6c38890de981ae92fb5eb50002d787171 not found: ID does not exist"
Sep 11 12:18:31.205094 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:31.205069    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"]
Sep 11 12:18:31.207458 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:31.207439    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af081c-845bc7cc88-4hc8h"]
Sep 11 12:18:32.069874 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:32.069833    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6093502b-ce5d-42aa-9c45-41fe039002c3" path="/var/lib/kubelet/pods/6093502b-ce5d-42aa-9c45-41fe039002c3/volumes"
Sep 11 12:18:48.391151 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:48.391113    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"]
Sep 11 12:18:48.391718 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:48.391380    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj" podUID="3cdf6e64-d037-4572-bfe5-72a763f0766d" containerName="maas-api" containerID="cri-o://6fcdd9ab6b5879df1d58a4516081dcf1ee145ed71b8890a873b26bb08a316da6" gracePeriod=30
Sep 11 12:18:48.641520 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:48.641457    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:18:48.744242 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:48.744209    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cdf6e64-d037-4572-bfe5-72a763f0766d-metrics-certs\") pod \"3cdf6e64-d037-4572-bfe5-72a763f0766d\" (UID: \"3cdf6e64-d037-4572-bfe5-72a763f0766d\") "
Sep 11 12:18:48.744422 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:48.744255    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cdf6e64-d037-4572-bfe5-72a763f0766d-openshift-service-ca\") pod \"3cdf6e64-d037-4572-bfe5-72a763f0766d\" (UID: \"3cdf6e64-d037-4572-bfe5-72a763f0766d\") "
Sep 11 12:18:48.744422 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:48.744310    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cdf6e64-d037-4572-bfe5-72a763f0766d-maas-api-tls\") pod \"3cdf6e64-d037-4572-bfe5-72a763f0766d\" (UID: \"3cdf6e64-d037-4572-bfe5-72a763f0766d\") "
Sep 11 12:18:48.744422 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:48.744349    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lmtg\" (UniqueName: \"kubernetes.io/projected/3cdf6e64-d037-4572-bfe5-72a763f0766d-kube-api-access-7lmtg\") pod \"3cdf6e64-d037-4572-bfe5-72a763f0766d\" (UID: \"3cdf6e64-d037-4572-bfe5-72a763f0766d\") "
Sep 11 12:18:48.744747 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:48.744722    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cdf6e64-d037-4572-bfe5-72a763f0766d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3cdf6e64-d037-4572-bfe5-72a763f0766d" (UID: "3cdf6e64-d037-4572-bfe5-72a763f0766d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:18:48.746562 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:48.746533    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cdf6e64-d037-4572-bfe5-72a763f0766d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3cdf6e64-d037-4572-bfe5-72a763f0766d" (UID: "3cdf6e64-d037-4572-bfe5-72a763f0766d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:18:48.746665 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:48.746559    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cdf6e64-d037-4572-bfe5-72a763f0766d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3cdf6e64-d037-4572-bfe5-72a763f0766d" (UID: "3cdf6e64-d037-4572-bfe5-72a763f0766d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:18:48.746665 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:48.746611    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cdf6e64-d037-4572-bfe5-72a763f0766d-kube-api-access-7lmtg" (OuterVolumeSpecName: "kube-api-access-7lmtg") pod "3cdf6e64-d037-4572-bfe5-72a763f0766d" (UID: "3cdf6e64-d037-4572-bfe5-72a763f0766d"). InnerVolumeSpecName "kube-api-access-7lmtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:18:48.845253 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:48.845225    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cdf6e64-d037-4572-bfe5-72a763f0766d-openshift-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:18:48.845253 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:48.845247    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cdf6e64-d037-4572-bfe5-72a763f0766d-maas-api-tls\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:18:48.845253 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:48.845256    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lmtg\" (UniqueName: \"kubernetes.io/projected/3cdf6e64-d037-4572-bfe5-72a763f0766d-kube-api-access-7lmtg\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:18:48.845453 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:48.845266    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cdf6e64-d037-4572-bfe5-72a763f0766d-metrics-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:18:49.271718 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:49.271680    2569 generic.go:356] "Generic (PLEG): container finished" podID="3cdf6e64-d037-4572-bfe5-72a763f0766d" containerID="6fcdd9ab6b5879df1d58a4516081dcf1ee145ed71b8890a873b26bb08a316da6" exitCode=0
Sep 11 12:18:49.271897 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:49.271725    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"
Sep 11 12:18:49.271897 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:49.271767    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj" event={"ID":"3cdf6e64-d037-4572-bfe5-72a763f0766d","Type":"ContainerDied","Data":"6fcdd9ab6b5879df1d58a4516081dcf1ee145ed71b8890a873b26bb08a316da6"}
Sep 11 12:18:49.271897 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:49.271816    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj" event={"ID":"3cdf6e64-d037-4572-bfe5-72a763f0766d","Type":"ContainerDied","Data":"cd1d6cda183f46d44a1d49b6b86ac37b09b052347c8dc33259f571dc84afe1ca"}
Sep 11 12:18:49.271897 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:49.271840    2569 scope.go:117] "RemoveContainer" containerID="6fcdd9ab6b5879df1d58a4516081dcf1ee145ed71b8890a873b26bb08a316da6"
Sep 11 12:18:49.281378 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:49.281361    2569 scope.go:117] "RemoveContainer" containerID="6fcdd9ab6b5879df1d58a4516081dcf1ee145ed71b8890a873b26bb08a316da6"
Sep 11 12:18:49.281667 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:18:49.281649    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fcdd9ab6b5879df1d58a4516081dcf1ee145ed71b8890a873b26bb08a316da6\": container with ID starting with 6fcdd9ab6b5879df1d58a4516081dcf1ee145ed71b8890a873b26bb08a316da6 not found: ID does not exist" containerID="6fcdd9ab6b5879df1d58a4516081dcf1ee145ed71b8890a873b26bb08a316da6"
Sep 11 12:18:49.281727 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:49.281677    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fcdd9ab6b5879df1d58a4516081dcf1ee145ed71b8890a873b26bb08a316da6"} err="failed to get container status \"6fcdd9ab6b5879df1d58a4516081dcf1ee145ed71b8890a873b26bb08a316da6\": rpc error: code = NotFound desc = could not find container \"6fcdd9ab6b5879df1d58a4516081dcf1ee145ed71b8890a873b26bb08a316da6\": container with ID starting with 6fcdd9ab6b5879df1d58a4516081dcf1ee145ed71b8890a873b26bb08a316da6 not found: ID does not exist"
Sep 11 12:18:49.297402 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:49.297382    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"]
Sep 11 12:18:49.303904 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:49.303882    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d844cc-787bfdb4d-9x6cj"]
Sep 11 12:18:50.070572 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:50.070544    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cdf6e64-d037-4572-bfe5-72a763f0766d" path="/var/lib/kubelet/pods/3cdf6e64-d037-4572-bfe5-72a763f0766d/volumes"
Sep 11 12:18:58.531123 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:58.531079    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c4fd66f5c-w44qc"]
Sep 11 12:18:58.531491 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:58.531347    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c4fd66f5c-w44qc" podUID="c72257ac-4829-4f36-8c0f-340082a19b2b" containerName="manager" containerID="cri-o://9f4397adbc7bc66d9be7d0a4a51d472b32c0f60fbfafdab235e9fcca1d099a15" gracePeriod=30
Sep 11 12:18:58.776219 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:58.776160    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4fd66f5c-w44qc"
Sep 11 12:18:58.931749 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:58.931724    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c72257ac-4829-4f36-8c0f-340082a19b2b-metrics-certs\") pod \"c72257ac-4829-4f36-8c0f-340082a19b2b\" (UID: \"c72257ac-4829-4f36-8c0f-340082a19b2b\") "
Sep 11 12:18:58.931913 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:58.931827    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78lns\" (UniqueName: \"kubernetes.io/projected/c72257ac-4829-4f36-8c0f-340082a19b2b-kube-api-access-78lns\") pod \"c72257ac-4829-4f36-8c0f-340082a19b2b\" (UID: \"c72257ac-4829-4f36-8c0f-340082a19b2b\") "
Sep 11 12:18:58.931913 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:58.931855    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c72257ac-4829-4f36-8c0f-340082a19b2b-webhook-cert\") pod \"c72257ac-4829-4f36-8c0f-340082a19b2b\" (UID: \"c72257ac-4829-4f36-8c0f-340082a19b2b\") "
Sep 11 12:18:58.933897 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:58.933861    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c72257ac-4829-4f36-8c0f-340082a19b2b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c72257ac-4829-4f36-8c0f-340082a19b2b" (UID: "c72257ac-4829-4f36-8c0f-340082a19b2b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:18:58.934020 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:58.933887    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c72257ac-4829-4f36-8c0f-340082a19b2b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c72257ac-4829-4f36-8c0f-340082a19b2b" (UID: "c72257ac-4829-4f36-8c0f-340082a19b2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:18:58.934020 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:58.933915    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72257ac-4829-4f36-8c0f-340082a19b2b-kube-api-access-78lns" (OuterVolumeSpecName: "kube-api-access-78lns") pod "c72257ac-4829-4f36-8c0f-340082a19b2b" (UID: "c72257ac-4829-4f36-8c0f-340082a19b2b"). InnerVolumeSpecName "kube-api-access-78lns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:18:59.033225 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:59.033167    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78lns\" (UniqueName: \"kubernetes.io/projected/c72257ac-4829-4f36-8c0f-340082a19b2b-kube-api-access-78lns\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:18:59.033225 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:59.033190    2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c72257ac-4829-4f36-8c0f-340082a19b2b-webhook-cert\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:18:59.033225 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:59.033199    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c72257ac-4829-4f36-8c0f-340082a19b2b-metrics-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:18:59.328519 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:59.328428    2569 generic.go:356] "Generic (PLEG): container finished" podID="c72257ac-4829-4f36-8c0f-340082a19b2b" containerID="9f4397adbc7bc66d9be7d0a4a51d472b32c0f60fbfafdab235e9fcca1d099a15" exitCode=0
Sep 11 12:18:59.328519 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:59.328499    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4fd66f5c-w44qc"
Sep 11 12:18:59.328707 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:59.328514    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4fd66f5c-w44qc" event={"ID":"c72257ac-4829-4f36-8c0f-340082a19b2b","Type":"ContainerDied","Data":"9f4397adbc7bc66d9be7d0a4a51d472b32c0f60fbfafdab235e9fcca1d099a15"}
Sep 11 12:18:59.328707 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:59.328603    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4fd66f5c-w44qc" event={"ID":"c72257ac-4829-4f36-8c0f-340082a19b2b","Type":"ContainerDied","Data":"9eea1e5e0fced4eb4e66edf7414d2a9535786ac5b43c73a049a96ebbb99ea783"}
Sep 11 12:18:59.328707 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:59.328632    2569 scope.go:117] "RemoveContainer" containerID="9f4397adbc7bc66d9be7d0a4a51d472b32c0f60fbfafdab235e9fcca1d099a15"
Sep 11 12:18:59.338507 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:59.338491    2569 scope.go:117] "RemoveContainer" containerID="9f4397adbc7bc66d9be7d0a4a51d472b32c0f60fbfafdab235e9fcca1d099a15"
Sep 11 12:18:59.338775 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:18:59.338757    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f4397adbc7bc66d9be7d0a4a51d472b32c0f60fbfafdab235e9fcca1d099a15\": container with ID starting with 9f4397adbc7bc66d9be7d0a4a51d472b32c0f60fbfafdab235e9fcca1d099a15 not found: ID does not exist" containerID="9f4397adbc7bc66d9be7d0a4a51d472b32c0f60fbfafdab235e9fcca1d099a15"
Sep 11 12:18:59.338848 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:59.338787    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f4397adbc7bc66d9be7d0a4a51d472b32c0f60fbfafdab235e9fcca1d099a15"} err="failed to get container status \"9f4397adbc7bc66d9be7d0a4a51d472b32c0f60fbfafdab235e9fcca1d099a15\": rpc error: code = NotFound desc = could not find container \"9f4397adbc7bc66d9be7d0a4a51d472b32c0f60fbfafdab235e9fcca1d099a15\": container with ID starting with 9f4397adbc7bc66d9be7d0a4a51d472b32c0f60fbfafdab235e9fcca1d099a15 not found: ID does not exist"
Sep 11 12:18:59.351463 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:59.351441    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c4fd66f5c-w44qc"]
Sep 11 12:18:59.353629 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:18:59.353610    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c4fd66f5c-w44qc"]
Sep 11 12:19:00.070079 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:19:00.070049    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c72257ac-4829-4f36-8c0f-340082a19b2b" path="/var/lib/kubelet/pods/c72257ac-4829-4f36-8c0f-340082a19b2b/volumes"
Sep 11 12:20:58.553308 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:20:58.553235    2569 scope.go:117] "RemoveContainer" containerID="00ed6b701d819b187806ec2abf6d502b6ef4c5f70b7f1f2fcb13065878260070"
Sep 11 12:22:58.261857 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:22:58.261831    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 12:22:58.270214 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:22:58.270191    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b6xv8_72619839-fc9b-42d3-8442-d14c24b423f5/ovn-acl-logging/0.log"
Sep 11 12:23:22.346653 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.346619    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"]
Sep 11 12:23:22.347023 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.346980    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7780da3-ec88-4e47-b90f-4bfc63ff060f" containerName="maas-api"
Sep 11 12:23:22.347023 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.346990    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7780da3-ec88-4e47-b90f-4bfc63ff060f" containerName="maas-api"
Sep 11 12:23:22.347023 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.347005    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c84368-cf8d-4dc1-918b-a79f3a9556cc" containerName="maas-api"
Sep 11 12:23:22.347023 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.347011    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c84368-cf8d-4dc1-918b-a79f3a9556cc" containerName="maas-api"
Sep 11 12:23:22.347023 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.347019    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6093502b-ce5d-42aa-9c45-41fe039002c3" containerName="maas-api"
Sep 11 12:23:22.347023 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.347024    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6093502b-ce5d-42aa-9c45-41fe039002c3" containerName="maas-api"
Sep 11 12:23:22.347221 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.347043    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c72257ac-4829-4f36-8c0f-340082a19b2b" containerName="manager"
Sep 11 12:23:22.347221 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.347048    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72257ac-4829-4f36-8c0f-340082a19b2b" containerName="manager"
Sep 11 12:23:22.347221 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.347054    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36b9c7a7-88f2-4498-8452-e6c517f192d9" containerName="main"
Sep 11 12:23:22.347221 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.347059    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="36b9c7a7-88f2-4498-8452-e6c517f192d9" containerName="main"
Sep 11 12:23:22.347221 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.347065    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cdf6e64-d037-4572-bfe5-72a763f0766d" containerName="maas-api"
Sep 11 12:23:22.347221 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.347070    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cdf6e64-d037-4572-bfe5-72a763f0766d" containerName="maas-api"
Sep 11 12:23:22.347221 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.347125    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4c84368-cf8d-4dc1-918b-a79f3a9556cc" containerName="maas-api"
Sep 11 12:23:22.347221 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.347133    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cdf6e64-d037-4572-bfe5-72a763f0766d" containerName="maas-api"
Sep 11 12:23:22.347221 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.347141    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6093502b-ce5d-42aa-9c45-41fe039002c3" containerName="maas-api"
Sep 11 12:23:22.347221 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.347148    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="36b9c7a7-88f2-4498-8452-e6c517f192d9" containerName="main"
Sep 11 12:23:22.347221 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.347154    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7780da3-ec88-4e47-b90f-4bfc63ff060f" containerName="maas-api"
Sep 11 12:23:22.347221 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.347160    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c72257ac-4829-4f36-8c0f-340082a19b2b" containerName="manager"
Sep 11 12:23:22.350136 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.350120    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.354163 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.354142    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e0808799-kserve-self-signed-certs\""
Sep 11 12:23:22.362571 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.362545    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"]
Sep 11 12:23:22.398118 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.398093    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-tmp-dir\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.398271 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.398133    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-home\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.398271 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.398153    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba5eeae2-c2b3-4421-822c-141e428f96f0-tls-certs\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.398271 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.398199    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-dshm\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.398404 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.398274    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-model-cache\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.398404 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.398301    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mptqv\" (UniqueName: \"kubernetes.io/projected/ba5eeae2-c2b3-4421-822c-141e428f96f0-kube-api-access-mptqv\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.499376 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.499347    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-home\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.499376 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.499378    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba5eeae2-c2b3-4421-822c-141e428f96f0-tls-certs\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.499614 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.499402    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-dshm\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.499614 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.499461    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-model-cache\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.499614 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.499495    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptqv\" (UniqueName: \"kubernetes.io/projected/ba5eeae2-c2b3-4421-822c-141e428f96f0-kube-api-access-mptqv\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.499614 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.499529    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-tmp-dir\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.499853 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.499826    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-home\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.499910 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.499874    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-tmp-dir\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.499963 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.499935    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-model-cache\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.501853 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.501831    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-dshm\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.501956 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.501937    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba5eeae2-c2b3-4421-822c-141e428f96f0-tls-certs\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.517903 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.517885    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mptqv\" (UniqueName: \"kubernetes.io/projected/ba5eeae2-c2b3-4421-822c-141e428f96f0-kube-api-access-mptqv\") pod \"e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") " pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.659151 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.659074    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:22.947963 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:22.947935    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"]
Sep 11 12:23:22.950392 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:23:22.950368    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba5eeae2_c2b3_4421_822c_141e428f96f0.slice/crio-af829f127bff882b261d27dd132eb9f5b0000606bb0cdd09ae5714ab455daaae WatchSource:0}: Error finding container af829f127bff882b261d27dd132eb9f5b0000606bb0cdd09ae5714ab455daaae: Status 404 returned error can't find the container with id af829f127bff882b261d27dd132eb9f5b0000606bb0cdd09ae5714ab455daaae
Sep 11 12:23:23.745904 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:23.745874    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt" event={"ID":"ba5eeae2-c2b3-4421-822c-141e428f96f0","Type":"ContainerStarted","Data":"4bff63b79f8e3f02909b980ffbdc6df13f73dfdaa25bc09b2c0be0d6745b2d86"}
Sep 11 12:23:23.746256 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:23.745912    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt" event={"ID":"ba5eeae2-c2b3-4421-822c-141e428f96f0","Type":"ContainerStarted","Data":"af829f127bff882b261d27dd132eb9f5b0000606bb0cdd09ae5714ab455daaae"}
Sep 11 12:23:23.770109 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:23.770062    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt" podStartSLOduration=1.7700456519999999 podStartE2EDuration="1.770045652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:23:23.768284227 +0000 UTC m=+2126.190152627" watchObservedRunningTime="2026-09-11 12:23:23.770045652 +0000 UTC m=+2126.191914032"
Sep 11 12:23:32.659633 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:32.659597    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:32.659633 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:32.659643    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:32.671956 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:32.671925    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:23:32.804862 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:23:32.804831    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:24:11.861060 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:11.861021    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"]
Sep 11 12:24:26.879793 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:26.879733    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt" podUID="ba5eeae2-c2b3-4421-822c-141e428f96f0" containerName="main" containerID="cri-o://4bff63b79f8e3f02909b980ffbdc6df13f73dfdaa25bc09b2c0be0d6745b2d86" gracePeriod=45
Sep 11 12:24:27.087569 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.087538    2569 generic.go:356] "Generic (PLEG): container finished" podID="ba5eeae2-c2b3-4421-822c-141e428f96f0" containerID="4bff63b79f8e3f02909b980ffbdc6df13f73dfdaa25bc09b2c0be0d6745b2d86" exitCode=0
Sep 11 12:24:27.087802 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.087647    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt" event={"ID":"ba5eeae2-c2b3-4421-822c-141e428f96f0","Type":"ContainerDied","Data":"4bff63b79f8e3f02909b980ffbdc6df13f73dfdaa25bc09b2c0be0d6745b2d86"}
Sep 11 12:24:27.125265 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.125244    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:24:27.259894 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.259871    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-home\") pod \"ba5eeae2-c2b3-4421-822c-141e428f96f0\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") "
Sep 11 12:24:27.260045 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.259916    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-tmp-dir\") pod \"ba5eeae2-c2b3-4421-822c-141e428f96f0\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") "
Sep 11 12:24:27.260045 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.259951    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-dshm\") pod \"ba5eeae2-c2b3-4421-822c-141e428f96f0\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") "
Sep 11 12:24:27.260045 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.259981    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-model-cache\") pod \"ba5eeae2-c2b3-4421-822c-141e428f96f0\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") "
Sep 11 12:24:27.260045 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.260031    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mptqv\" (UniqueName: \"kubernetes.io/projected/ba5eeae2-c2b3-4421-822c-141e428f96f0-kube-api-access-mptqv\") pod \"ba5eeae2-c2b3-4421-822c-141e428f96f0\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") "
Sep 11 12:24:27.260273 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.260049    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba5eeae2-c2b3-4421-822c-141e428f96f0-tls-certs\") pod \"ba5eeae2-c2b3-4421-822c-141e428f96f0\" (UID: \"ba5eeae2-c2b3-4421-822c-141e428f96f0\") "
Sep 11 12:24:27.260273 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.260151    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-home" (OuterVolumeSpecName: "home") pod "ba5eeae2-c2b3-4421-822c-141e428f96f0" (UID: "ba5eeae2-c2b3-4421-822c-141e428f96f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:24:27.260273 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.260233    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba5eeae2-c2b3-4421-822c-141e428f96f0" (UID: "ba5eeae2-c2b3-4421-822c-141e428f96f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:24:27.260423 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.260283    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba5eeae2-c2b3-4421-822c-141e428f96f0" (UID: "ba5eeae2-c2b3-4421-822c-141e428f96f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:24:27.260423 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.260368    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-model-cache\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:24:27.260423 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.260380    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-home\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:24:27.260423 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.260390    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-tmp-dir\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:24:27.261952 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.261926    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-dshm" (OuterVolumeSpecName: "dshm") pod "ba5eeae2-c2b3-4421-822c-141e428f96f0" (UID: "ba5eeae2-c2b3-4421-822c-141e428f96f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:24:27.262084 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.262061    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba5eeae2-c2b3-4421-822c-141e428f96f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba5eeae2-c2b3-4421-822c-141e428f96f0" (UID: "ba5eeae2-c2b3-4421-822c-141e428f96f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:24:27.262136 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.262111    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5eeae2-c2b3-4421-822c-141e428f96f0-kube-api-access-mptqv" (OuterVolumeSpecName: "kube-api-access-mptqv") pod "ba5eeae2-c2b3-4421-822c-141e428f96f0" (UID: "ba5eeae2-c2b3-4421-822c-141e428f96f0"). InnerVolumeSpecName "kube-api-access-mptqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:24:27.361403 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.361380    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mptqv\" (UniqueName: \"kubernetes.io/projected/ba5eeae2-c2b3-4421-822c-141e428f96f0-kube-api-access-mptqv\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:24:27.361403 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.361401    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba5eeae2-c2b3-4421-822c-141e428f96f0-tls-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:24:27.361531 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:27.361410    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba5eeae2-c2b3-4421-822c-141e428f96f0-dshm\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:24:28.094250 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:28.094221    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"
Sep 11 12:24:28.094250 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:28.094236    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt" event={"ID":"ba5eeae2-c2b3-4421-822c-141e428f96f0","Type":"ContainerDied","Data":"af829f127bff882b261d27dd132eb9f5b0000606bb0cdd09ae5714ab455daaae"}
Sep 11 12:24:28.094657 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:28.094279    2569 scope.go:117] "RemoveContainer" containerID="4bff63b79f8e3f02909b980ffbdc6df13f73dfdaa25bc09b2c0be0d6745b2d86"
Sep 11 12:24:28.112245 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:28.112223    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"]
Sep 11 12:24:28.115943 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:28.115921    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e0808799-kserve-bf4dc577f-xwcgt"]
Sep 11 12:24:30.070209 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:30.070180    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba5eeae2-c2b3-4421-822c-141e428f96f0" path="/var/lib/kubelet/pods/ba5eeae2-c2b3-4421-822c-141e428f96f0/volumes"
Sep 11 12:24:32.766029 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:32.765995    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"]
Sep 11 12:24:32.766565 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:32.766546    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba5eeae2-c2b3-4421-822c-141e428f96f0" containerName="main"
Sep 11 12:24:32.766678 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:32.766568    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba5eeae2-c2b3-4421-822c-141e428f96f0" containerName="main"
Sep 11 12:24:32.766738 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:32.766712    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba5eeae2-c2b3-4421-822c-141e428f96f0" containerName="main"
Sep 11 12:24:32.771363 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:32.771342    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:32.774942 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:32.774922    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 12:24:32.775183 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:32.775162    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rrxjh\""
Sep 11 12:24:32.775270 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:32.775250    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 12:24:32.781385 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:32.781364    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"]
Sep 11 12:24:32.913907 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:32.913876    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad0328d6-57de-4c7b-9129-abf4d17dc888-metrics-certs\") pod \"maas-api-669df8b46-bb9mr\" (UID: \"ad0328d6-57de-4c7b-9129-abf4d17dc888\") " pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:32.913907 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:32.913910    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htkkj\" (UniqueName: \"kubernetes.io/projected/ad0328d6-57de-4c7b-9129-abf4d17dc888-kube-api-access-htkkj\") pod \"maas-api-669df8b46-bb9mr\" (UID: \"ad0328d6-57de-4c7b-9129-abf4d17dc888\") " pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:32.914128 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:32.913954    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad0328d6-57de-4c7b-9129-abf4d17dc888-maas-api-tls\") pod \"maas-api-669df8b46-bb9mr\" (UID: \"ad0328d6-57de-4c7b-9129-abf4d17dc888\") " pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:32.914128 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:32.914047    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0328d6-57de-4c7b-9129-abf4d17dc888-openshift-service-ca\") pod \"maas-api-669df8b46-bb9mr\" (UID: \"ad0328d6-57de-4c7b-9129-abf4d17dc888\") " pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:33.014383 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:33.014357    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad0328d6-57de-4c7b-9129-abf4d17dc888-metrics-certs\") pod \"maas-api-669df8b46-bb9mr\" (UID: \"ad0328d6-57de-4c7b-9129-abf4d17dc888\") " pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:33.014501 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:33.014386    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htkkj\" (UniqueName: \"kubernetes.io/projected/ad0328d6-57de-4c7b-9129-abf4d17dc888-kube-api-access-htkkj\") pod \"maas-api-669df8b46-bb9mr\" (UID: \"ad0328d6-57de-4c7b-9129-abf4d17dc888\") " pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:33.014501 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:33.014425    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad0328d6-57de-4c7b-9129-abf4d17dc888-maas-api-tls\") pod \"maas-api-669df8b46-bb9mr\" (UID: \"ad0328d6-57de-4c7b-9129-abf4d17dc888\") " pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:33.014501 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:33.014444    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0328d6-57de-4c7b-9129-abf4d17dc888-openshift-service-ca\") pod \"maas-api-669df8b46-bb9mr\" (UID: \"ad0328d6-57de-4c7b-9129-abf4d17dc888\") " pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:33.015057 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:33.015025    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0328d6-57de-4c7b-9129-abf4d17dc888-openshift-service-ca\") pod \"maas-api-669df8b46-bb9mr\" (UID: \"ad0328d6-57de-4c7b-9129-abf4d17dc888\") " pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:33.016949 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:33.016906    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad0328d6-57de-4c7b-9129-abf4d17dc888-metrics-certs\") pod \"maas-api-669df8b46-bb9mr\" (UID: \"ad0328d6-57de-4c7b-9129-abf4d17dc888\") " pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:33.017017 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:33.017001    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad0328d6-57de-4c7b-9129-abf4d17dc888-maas-api-tls\") pod \"maas-api-669df8b46-bb9mr\" (UID: \"ad0328d6-57de-4c7b-9129-abf4d17dc888\") " pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:33.023038 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:33.023012    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htkkj\" (UniqueName: \"kubernetes.io/projected/ad0328d6-57de-4c7b-9129-abf4d17dc888-kube-api-access-htkkj\") pod \"maas-api-669df8b46-bb9mr\" (UID: \"ad0328d6-57de-4c7b-9129-abf4d17dc888\") " pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:33.083144 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:33.083120    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:33.370961 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:33.370931    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"]
Sep 11 12:24:33.373719 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:24:33.373696    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0328d6_57de_4c7b_9129_abf4d17dc888.slice/crio-d2684f790f6ba53c4fcb10a5bae45fa5d3fea7f632dd90804fa5a10717f4bbaa WatchSource:0}: Error finding container d2684f790f6ba53c4fcb10a5bae45fa5d3fea7f632dd90804fa5a10717f4bbaa: Status 404 returned error can't find the container with id d2684f790f6ba53c4fcb10a5bae45fa5d3fea7f632dd90804fa5a10717f4bbaa
Sep 11 12:24:33.375125 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:33.375101    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:24:34.132209 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:34.132123    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr" event={"ID":"ad0328d6-57de-4c7b-9129-abf4d17dc888","Type":"ContainerStarted","Data":"c54f339aca6cd26e68dc5a6a1aa196b447d1dc281dca9418669f36e0f5b693d0"}
Sep 11 12:24:34.132209 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:34.132173    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr" event={"ID":"ad0328d6-57de-4c7b-9129-abf4d17dc888","Type":"ContainerStarted","Data":"d2684f790f6ba53c4fcb10a5bae45fa5d3fea7f632dd90804fa5a10717f4bbaa"}
Sep 11 12:24:34.132209 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:34.132198    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:34.151434 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:34.151377    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr" podStartSLOduration=1.70130392 podStartE2EDuration="2.151356794s" totalImagesPullingTime="450.052874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:24:34.151192017 +0000 UTC m=+2196.573060421" watchObservedRunningTime="2026-09-11 12:24:34.151356794 +0000 UTC m=+2196.573225176"
Sep 11 12:24:40.142982 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:40.142950    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:40.293247 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:40.293214    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"]
Sep 11 12:24:40.293483 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:40.293435    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr" podUID="ad0328d6-57de-4c7b-9129-abf4d17dc888" containerName="maas-api" containerID="cri-o://c54f339aca6cd26e68dc5a6a1aa196b447d1dc281dca9418669f36e0f5b693d0" gracePeriod=30
Sep 11 12:24:40.541468 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:40.541440    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:40.681280 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:40.681202    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htkkj\" (UniqueName: \"kubernetes.io/projected/ad0328d6-57de-4c7b-9129-abf4d17dc888-kube-api-access-htkkj\") pod \"ad0328d6-57de-4c7b-9129-abf4d17dc888\" (UID: \"ad0328d6-57de-4c7b-9129-abf4d17dc888\") "
Sep 11 12:24:40.681280 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:40.681277    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad0328d6-57de-4c7b-9129-abf4d17dc888-maas-api-tls\") pod \"ad0328d6-57de-4c7b-9129-abf4d17dc888\" (UID: \"ad0328d6-57de-4c7b-9129-abf4d17dc888\") "
Sep 11 12:24:40.681509 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:40.681326    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0328d6-57de-4c7b-9129-abf4d17dc888-openshift-service-ca\") pod \"ad0328d6-57de-4c7b-9129-abf4d17dc888\" (UID: \"ad0328d6-57de-4c7b-9129-abf4d17dc888\") "
Sep 11 12:24:40.681509 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:40.681350    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad0328d6-57de-4c7b-9129-abf4d17dc888-metrics-certs\") pod \"ad0328d6-57de-4c7b-9129-abf4d17dc888\" (UID: \"ad0328d6-57de-4c7b-9129-abf4d17dc888\") "
Sep 11 12:24:40.681683 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:40.681660    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0328d6-57de-4c7b-9129-abf4d17dc888-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ad0328d6-57de-4c7b-9129-abf4d17dc888" (UID: "ad0328d6-57de-4c7b-9129-abf4d17dc888"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:24:40.683497 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:40.683470    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0328d6-57de-4c7b-9129-abf4d17dc888-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ad0328d6-57de-4c7b-9129-abf4d17dc888" (UID: "ad0328d6-57de-4c7b-9129-abf4d17dc888"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:24:40.683497 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:40.683489    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0328d6-57de-4c7b-9129-abf4d17dc888-kube-api-access-htkkj" (OuterVolumeSpecName: "kube-api-access-htkkj") pod "ad0328d6-57de-4c7b-9129-abf4d17dc888" (UID: "ad0328d6-57de-4c7b-9129-abf4d17dc888"). InnerVolumeSpecName "kube-api-access-htkkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:24:40.683673 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:40.683547    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0328d6-57de-4c7b-9129-abf4d17dc888-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ad0328d6-57de-4c7b-9129-abf4d17dc888" (UID: "ad0328d6-57de-4c7b-9129-abf4d17dc888"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:24:40.782579 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:40.782555    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad0328d6-57de-4c7b-9129-abf4d17dc888-maas-api-tls\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:24:40.782579 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:40.782578    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0328d6-57de-4c7b-9129-abf4d17dc888-openshift-service-ca\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:24:40.782579 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:40.782598    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad0328d6-57de-4c7b-9129-abf4d17dc888-metrics-certs\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:24:40.782756 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:40.782606    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htkkj\" (UniqueName: \"kubernetes.io/projected/ad0328d6-57de-4c7b-9129-abf4d17dc888-kube-api-access-htkkj\") on node \"ip-10-0-143-220.ec2.internal\" DevicePath \"\""
Sep 11 12:24:41.171387 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:41.171356    2569 generic.go:356] "Generic (PLEG): container finished" podID="ad0328d6-57de-4c7b-9129-abf4d17dc888" containerID="c54f339aca6cd26e68dc5a6a1aa196b447d1dc281dca9418669f36e0f5b693d0" exitCode=0
Sep 11 12:24:41.171849 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:41.171393    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"
Sep 11 12:24:41.171849 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:41.171436    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr" event={"ID":"ad0328d6-57de-4c7b-9129-abf4d17dc888","Type":"ContainerDied","Data":"c54f339aca6cd26e68dc5a6a1aa196b447d1dc281dca9418669f36e0f5b693d0"}
Sep 11 12:24:41.171849 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:41.171487    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-669df8b46-bb9mr" event={"ID":"ad0328d6-57de-4c7b-9129-abf4d17dc888","Type":"ContainerDied","Data":"d2684f790f6ba53c4fcb10a5bae45fa5d3fea7f632dd90804fa5a10717f4bbaa"}
Sep 11 12:24:41.171849 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:41.171509    2569 scope.go:117] "RemoveContainer" containerID="c54f339aca6cd26e68dc5a6a1aa196b447d1dc281dca9418669f36e0f5b693d0"
Sep 11 12:24:41.180940 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:41.180925    2569 scope.go:117] "RemoveContainer" containerID="c54f339aca6cd26e68dc5a6a1aa196b447d1dc281dca9418669f36e0f5b693d0"
Sep 11 12:24:41.181221 ip-10-0-143-220 kubenswrapper[2569]: E0911 12:24:41.181204    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c54f339aca6cd26e68dc5a6a1aa196b447d1dc281dca9418669f36e0f5b693d0\": container with ID starting with c54f339aca6cd26e68dc5a6a1aa196b447d1dc281dca9418669f36e0f5b693d0 not found: ID does not exist" containerID="c54f339aca6cd26e68dc5a6a1aa196b447d1dc281dca9418669f36e0f5b693d0"
Sep 11 12:24:41.181276 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:41.181230    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54f339aca6cd26e68dc5a6a1aa196b447d1dc281dca9418669f36e0f5b693d0"} err="failed to get container status \"c54f339aca6cd26e68dc5a6a1aa196b447d1dc281dca9418669f36e0f5b693d0\": rpc error: code = NotFound desc = could not find container \"c54f339aca6cd26e68dc5a6a1aa196b447d1dc281dca9418669f36e0f5b693d0\": container with ID starting with c54f339aca6cd26e68dc5a6a1aa196b447d1dc281dca9418669f36e0f5b693d0 not found: ID does not exist"
Sep 11 12:24:41.194798 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:41.194776    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"]
Sep 11 12:24:41.199366 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:41.199346    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-669df8b46-bb9mr"]
Sep 11 12:24:42.069922 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:24:42.069878    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad0328d6-57de-4c7b-9129-abf4d17dc888" path="/var/lib/kubelet/pods/ad0328d6-57de-4c7b-9129-abf4d17dc888/volumes"
Sep 11 12:25:02.976005 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:02.975975    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55b4c84b95-wvdvm_712d2398-1e74-4859-9466-587d71ae859c/authorino/0.log"
Sep 11 12:25:09.589566 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:09.589538    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fb8846894-krlxl_468a44cb-07a8-4625-bed8-ba64db6c29d7/copy-manifests/0.log"
Sep 11 12:25:09.597000 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:09.596975    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fb8846894-krlxl_468a44cb-07a8-4625-bed8-ba64db6c29d7/manager/0.log"
Sep 11 12:25:10.597691 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:10.597657    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-t2bhm_3c7a3895-ea08-4b99-a07e-21e7fb8667a6/postgres/0.log"
Sep 11 12:25:11.450037 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:11.449971    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55b4c84b95-wvdvm_712d2398-1e74-4859-9466-587d71ae859c/authorino/0.log"
Sep 11 12:25:12.795282 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:12.795249    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57dff8dc5-hjn5h_047ffb0a-9d8a-42c1-a8ff-83f840a4bf80/kube-auth-proxy/0.log"
Sep 11 12:25:13.195119 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:13.195045    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-5bqf9_c1595fd2-0403-487d-a7ea-3ae02b1309a7/payload-pre-processing/0.log"
Sep 11 12:25:13.822290 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:13.822262    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-f4rvh_03c3f321-c61d-47e5-8a7a-0ead31f5e6dd/main/0.log"
Sep 11 12:25:14.224833 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:14.224752    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-cj6n2_320bb8fb-1a01-4e74-804d-b53916b0458e/main/0.log"
Sep 11 12:25:14.617796 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:14.617757    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-pr4zc_43ba295e-5044-4635-8a7d-0550dae17127/main/0.log"
Sep 11 12:25:20.004788 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:20.004753    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vdhzr/must-gather-2vlcq"]
Sep 11 12:25:20.005179 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:20.005165    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad0328d6-57de-4c7b-9129-abf4d17dc888" containerName="maas-api"
Sep 11 12:25:20.005226 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:20.005181    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad0328d6-57de-4c7b-9129-abf4d17dc888" containerName="maas-api"
Sep 11 12:25:20.005265 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:20.005248    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad0328d6-57de-4c7b-9129-abf4d17dc888" containerName="maas-api"
Sep 11 12:25:20.008221 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:20.008199    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vdhzr/must-gather-2vlcq"
Sep 11 12:25:20.010936 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:20.010909    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vdhzr\"/\"kube-root-ca.crt\""
Sep 11 12:25:20.011124 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:20.011104    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vdhzr\"/\"default-dockercfg-m29sz\""
Sep 11 12:25:20.011834 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:20.011814    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vdhzr\"/\"openshift-service-ca.crt\""
Sep 11 12:25:20.019971 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:20.019948    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vdhzr/must-gather-2vlcq"]
Sep 11 12:25:20.113441 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:20.113412    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aac1ba88-62c9-40ad-88a2-eed3c85648d6-must-gather-output\") pod \"must-gather-2vlcq\" (UID: \"aac1ba88-62c9-40ad-88a2-eed3c85648d6\") " pod="openshift-must-gather-vdhzr/must-gather-2vlcq"
Sep 11 12:25:20.113635 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:20.113451    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvl9q\" (UniqueName: \"kubernetes.io/projected/aac1ba88-62c9-40ad-88a2-eed3c85648d6-kube-api-access-nvl9q\") pod \"must-gather-2vlcq\" (UID: \"aac1ba88-62c9-40ad-88a2-eed3c85648d6\") " pod="openshift-must-gather-vdhzr/must-gather-2vlcq"
Sep 11 12:25:20.214192 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:20.214158    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aac1ba88-62c9-40ad-88a2-eed3c85648d6-must-gather-output\") pod \"must-gather-2vlcq\" (UID: \"aac1ba88-62c9-40ad-88a2-eed3c85648d6\") " pod="openshift-must-gather-vdhzr/must-gather-2vlcq"
Sep 11 12:25:20.214192 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:20.214195    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvl9q\" (UniqueName: \"kubernetes.io/projected/aac1ba88-62c9-40ad-88a2-eed3c85648d6-kube-api-access-nvl9q\") pod \"must-gather-2vlcq\" (UID: \"aac1ba88-62c9-40ad-88a2-eed3c85648d6\") " pod="openshift-must-gather-vdhzr/must-gather-2vlcq"
Sep 11 12:25:20.214506 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:20.214487    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aac1ba88-62c9-40ad-88a2-eed3c85648d6-must-gather-output\") pod \"must-gather-2vlcq\" (UID: \"aac1ba88-62c9-40ad-88a2-eed3c85648d6\") " pod="openshift-must-gather-vdhzr/must-gather-2vlcq"
Sep 11 12:25:20.223773 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:20.223751    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvl9q\" (UniqueName: \"kubernetes.io/projected/aac1ba88-62c9-40ad-88a2-eed3c85648d6-kube-api-access-nvl9q\") pod \"must-gather-2vlcq\" (UID: \"aac1ba88-62c9-40ad-88a2-eed3c85648d6\") " pod="openshift-must-gather-vdhzr/must-gather-2vlcq"
Sep 11 12:25:20.317852 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:20.317789    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vdhzr/must-gather-2vlcq"
Sep 11 12:25:20.591901 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:20.591875    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vdhzr/must-gather-2vlcq"]
Sep 11 12:25:20.593575 ip-10-0-143-220 kubenswrapper[2569]: W0911 12:25:20.593539    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaac1ba88_62c9_40ad_88a2_eed3c85648d6.slice/crio-5cf2362f6c60b3bd007b30d4db58d101e1c6c4bf6d38525dd8f4d0d6379e0498 WatchSource:0}: Error finding container 5cf2362f6c60b3bd007b30d4db58d101e1c6c4bf6d38525dd8f4d0d6379e0498: Status 404 returned error can't find the container with id 5cf2362f6c60b3bd007b30d4db58d101e1c6c4bf6d38525dd8f4d0d6379e0498
Sep 11 12:25:21.394227 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:21.394195    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vdhzr/must-gather-2vlcq" event={"ID":"aac1ba88-62c9-40ad-88a2-eed3c85648d6","Type":"ContainerStarted","Data":"5cf2362f6c60b3bd007b30d4db58d101e1c6c4bf6d38525dd8f4d0d6379e0498"}
Sep 11 12:25:22.404328 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:22.404286    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vdhzr/must-gather-2vlcq" event={"ID":"aac1ba88-62c9-40ad-88a2-eed3c85648d6","Type":"ContainerStarted","Data":"6e0f165c8aff1788e64d90fb50309f7803e478f855a01cbe8515a32fbf846531"}
Sep 11 12:25:22.405158 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:22.404344    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vdhzr/must-gather-2vlcq" event={"ID":"aac1ba88-62c9-40ad-88a2-eed3c85648d6","Type":"ContainerStarted","Data":"7f78aa87de27b89bba0b8d7ac1d30bedbe565d663c49770921a4f8b8e8fe28bb"}
Sep 11 12:25:22.424065 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:22.424002    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vdhzr/must-gather-2vlcq" podStartSLOduration=2.708263318 podStartE2EDuration="3.423979491s" totalImagesPullingTime="715.716173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:25:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:25:22.421319261 +0000 UTC m=+2244.843187641" watchObservedRunningTime="2026-09-11 12:25:22.423979491 +0000 UTC m=+2244.845847875"
Sep 11 12:25:23.164762 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:23.164732    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sm98f_584f8d60-a99b-4ab3-acbe-c5de6343284f/global-pull-secret-syncer/0.log"
Sep 11 12:25:23.333822 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:23.333790    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vkfwj_8bb6ad09-b9bd-41ef-bf5d-2ae5a3c28aff/konnectivity-agent/0.log"
Sep 11 12:25:23.414153 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:23.414124    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-220.ec2.internal_52c830ce49e31490f686af3123732f5d/haproxy/0.log"
Sep 11 12:25:27.592970 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:27.592875    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55b4c84b95-wvdvm_712d2398-1e74-4859-9466-587d71ae859c/authorino/0.log"
Sep 11 12:25:29.371321 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:29.371284    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aebf0f41-2fa7-431b-b486-752354be0f44/alertmanager/0.log"
Sep 11 12:25:29.403308 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:29.403266    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aebf0f41-2fa7-431b-b486-752354be0f44/config-reloader/0.log"
Sep 11 12:25:29.442493 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:29.442464    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aebf0f41-2fa7-431b-b486-752354be0f44/kube-rbac-proxy-web/0.log"
Sep 11 12:25:29.503309 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:29.503255    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aebf0f41-2fa7-431b-b486-752354be0f44/kube-rbac-proxy/0.log"
Sep 11 12:25:29.540870 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:29.540838    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aebf0f41-2fa7-431b-b486-752354be0f44/kube-rbac-proxy-metric/0.log"
Sep 11 12:25:29.569380 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:29.569290    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aebf0f41-2fa7-431b-b486-752354be0f44/prom-label-proxy/0.log"
Sep 11 12:25:29.598173 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:29.598139    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aebf0f41-2fa7-431b-b486-752354be0f44/init-config-reloader/0.log"
Sep 11 12:25:29.650397 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:29.650315    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-vdr7c_c1a888c2-3145-46ea-90b4-d026de4a6ad0/cluster-monitoring-operator/0.log"
Sep 11 12:25:29.680507 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:29.680475    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-97kqm_0902197e-f7cd-43ba-b12e-243318ac4fd0/kube-state-metrics/0.log"
Sep 11 12:25:29.703842 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:29.703808    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-97kqm_0902197e-f7cd-43ba-b12e-243318ac4fd0/kube-rbac-proxy-main/0.log"
Sep 11 12:25:29.732254 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:29.732224    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-97kqm_0902197e-f7cd-43ba-b12e-243318ac4fd0/kube-rbac-proxy-self/0.log"
Sep 11 12:25:29.817160 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:29.817129    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-wbfgd_b82fa9c1-56f0-4167-a128-dc6948db6614/monitoring-plugin/0.log"
Sep 11 12:25:29.855958 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:29.855925    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2sql7_76c0c6c7-2fed-4522-8797-11bc095ce67e/node-exporter/0.log"
Sep 11 12:25:29.886954 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:29.886923    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2sql7_76c0c6c7-2fed-4522-8797-11bc095ce67e/kube-rbac-proxy/0.log"
Sep 11 12:25:29.913998 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:29.913912    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2sql7_76c0c6c7-2fed-4522-8797-11bc095ce67e/init-textfile/0.log"
Sep 11 12:25:30.470265 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:30.470180    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-2snb2_b3231f7c-5ad1-4376-9bb2-3dec096cb340/prometheus-operator/0.log"
Sep 11 12:25:30.506060 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:30.506026    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-2snb2_b3231f7c-5ad1-4376-9bb2-3dec096cb340/kube-rbac-proxy/0.log"
Sep 11 12:25:30.573741 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:30.573709    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7669fb6996-656pl_dbff4b49-3525-46aa-a15c-be05c0e439cc/telemeter-client/0.log"
Sep 11 12:25:30.603869 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:30.603840    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7669fb6996-656pl_dbff4b49-3525-46aa-a15c-be05c0e439cc/reload/0.log"
Sep 11 12:25:30.634473 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:30.634444    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7669fb6996-656pl_dbff4b49-3525-46aa-a15c-be05c0e439cc/kube-rbac-proxy/0.log"
Sep 11 12:25:32.991784 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:32.991747    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"]
Sep 11 12:25:33.000514 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:32.998687    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:33.012056 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.012017    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"]
Sep 11 12:25:33.166625 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.166567    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68075bf8-6358-470b-8d43-38dea60b6ac5-lib-modules\") pod \"perf-node-gather-daemonset-5txww\" (UID: \"68075bf8-6358-470b-8d43-38dea60b6ac5\") " pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:33.166825 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.166742    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68075bf8-6358-470b-8d43-38dea60b6ac5-sys\") pod \"perf-node-gather-daemonset-5txww\" (UID: \"68075bf8-6358-470b-8d43-38dea60b6ac5\") " pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:33.166825 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.166785    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68075bf8-6358-470b-8d43-38dea60b6ac5-podres\") pod \"perf-node-gather-daemonset-5txww\" (UID: \"68075bf8-6358-470b-8d43-38dea60b6ac5\") " pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:33.166825 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.166817    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68075bf8-6358-470b-8d43-38dea60b6ac5-proc\") pod \"perf-node-gather-daemonset-5txww\" (UID: \"68075bf8-6358-470b-8d43-38dea60b6ac5\") " pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:33.167037 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.166894    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m9q6\" (UniqueName: \"kubernetes.io/projected/68075bf8-6358-470b-8d43-38dea60b6ac5-kube-api-access-4m9q6\") pod \"perf-node-gather-daemonset-5txww\" (UID: \"68075bf8-6358-470b-8d43-38dea60b6ac5\") " pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:33.267990 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.267942    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68075bf8-6358-470b-8d43-38dea60b6ac5-sys\") pod \"perf-node-gather-daemonset-5txww\" (UID: \"68075bf8-6358-470b-8d43-38dea60b6ac5\") " pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:33.268192 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.268008    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68075bf8-6358-470b-8d43-38dea60b6ac5-podres\") pod \"perf-node-gather-daemonset-5txww\" (UID: \"68075bf8-6358-470b-8d43-38dea60b6ac5\") " pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:33.268192 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.268045    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68075bf8-6358-470b-8d43-38dea60b6ac5-proc\") pod \"perf-node-gather-daemonset-5txww\" (UID: \"68075bf8-6358-470b-8d43-38dea60b6ac5\") " pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:33.268192 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.268112    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m9q6\" (UniqueName: \"kubernetes.io/projected/68075bf8-6358-470b-8d43-38dea60b6ac5-kube-api-access-4m9q6\") pod \"perf-node-gather-daemonset-5txww\" (UID: \"68075bf8-6358-470b-8d43-38dea60b6ac5\") " pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:33.268192 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.268140    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68075bf8-6358-470b-8d43-38dea60b6ac5-lib-modules\") pod \"perf-node-gather-daemonset-5txww\" (UID: \"68075bf8-6358-470b-8d43-38dea60b6ac5\") " pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:33.268457 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.268367    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68075bf8-6358-470b-8d43-38dea60b6ac5-lib-modules\") pod \"perf-node-gather-daemonset-5txww\" (UID: \"68075bf8-6358-470b-8d43-38dea60b6ac5\") " pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:33.268457 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.268428    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68075bf8-6358-470b-8d43-38dea60b6ac5-sys\") pod \"perf-node-gather-daemonset-5txww\" (UID: \"68075bf8-6358-470b-8d43-38dea60b6ac5\") " pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:33.268573 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.268502    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68075bf8-6358-470b-8d43-38dea60b6ac5-podres\") pod \"perf-node-gather-daemonset-5txww\" (UID: \"68075bf8-6358-470b-8d43-38dea60b6ac5\") " pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:33.268573 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.268545    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68075bf8-6358-470b-8d43-38dea60b6ac5-proc\") pod \"perf-node-gather-daemonset-5txww\" (UID: \"68075bf8-6358-470b-8d43-38dea60b6ac5\") " pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:33.277971 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.277944    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m9q6\" (UniqueName: \"kubernetes.io/projected/68075bf8-6358-470b-8d43-38dea60b6ac5-kube-api-access-4m9q6\") pod \"perf-node-gather-daemonset-5txww\" (UID: \"68075bf8-6358-470b-8d43-38dea60b6ac5\") " pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:33.313045 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.313010    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:33.520398 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.520310    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7f5bd46c46-8q5kr_7841f7e9-a8e7-4343-b973-50e696a373c1/console/0.log"
Sep 11 12:25:33.562235 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.562182    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-jb96z_efb5b029-a20f-4b03-8814-4069153d2906/download-server/0.log"
Sep 11 12:25:33.621658 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:33.621622    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"]
Sep 11 12:25:34.253270 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:34.253231    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-7ps58_5b48c153-e731-4f3f-a4e6-254c18c4af63/volume-data-source-validator/0.log"
Sep 11 12:25:34.513655 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:34.513558    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww" event={"ID":"68075bf8-6358-470b-8d43-38dea60b6ac5","Type":"ContainerStarted","Data":"4186fff89b3ccd146ff03e80f9e4079b88dc3ed9482716f4f7a4dbeb7e02188f"}
Sep 11 12:25:34.513813 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:34.513662    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww" event={"ID":"68075bf8-6358-470b-8d43-38dea60b6ac5","Type":"ContainerStarted","Data":"9b6c10e7b765fc6c09e7990df0298289d7b8a72f7ff99dca0f3a33da8c15c793"}
Sep 11 12:25:34.514546 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:34.514522    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:34.532256 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:34.532194    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww" podStartSLOduration=2.532172402 podStartE2EDuration="2.532172402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:25:34.530244385 +0000 UTC m=+2256.952112796" watchObservedRunningTime="2026-09-11 12:25:34.532172402 +0000 UTC m=+2256.954040782"
Sep 11 12:25:35.400441 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:35.400405    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hmj9n_89c76d6c-b4b3-442a-8081-0ec81e5894a6/dns/0.log"
Sep 11 12:25:35.423847 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:35.423773    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hmj9n_89c76d6c-b4b3-442a-8081-0ec81e5894a6/kube-rbac-proxy/0.log"
Sep 11 12:25:35.559980 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:35.559950    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vk54k_714f5e6f-c22d-48de-b545-6be2e58968b0/dns-node-resolver/0.log"
Sep 11 12:25:36.097768 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:36.097725    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c869f9486-nt2v4_da3036fd-d0f4-44b4-876e-be8d5971886e/registry/0.log"
Sep 11 12:25:36.148806 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:36.148771    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fjbdx_96f46a0c-d972-4490-a888-ca0f79174b50/node-ca/0.log"
Sep 11 12:25:37.609124 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:37.609093    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57dff8dc5-hjn5h_047ffb0a-9d8a-42c1-a8ff-83f840a4bf80/kube-auth-proxy/0.log"
Sep 11 12:25:37.727529 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:37.727491    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-5bqf9_c1595fd2-0403-487d-a7ea-3ae02b1309a7/payload-pre-processing/0.log"
Sep 11 12:25:38.768159 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:38.768125    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c4qfd_775678cb-ae06-458a-a9f1-1ab4b5f48928/serve-healthcheck-canary/0.log"
Sep 11 12:25:39.338869 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:39.338843    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-859g8_8592c40b-2ab2-47ee-872b-4aa604fe5b36/kube-rbac-proxy/0.log"
Sep 11 12:25:39.360743 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:39.360705    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-859g8_8592c40b-2ab2-47ee-872b-4aa604fe5b36/exporter/0.log"
Sep 11 12:25:39.384242 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:39.384214    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-859g8_8592c40b-2ab2-47ee-872b-4aa604fe5b36/extractor/0.log"
Sep 11 12:25:41.540959 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:41.540924    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vdhzr/perf-node-gather-daemonset-5txww"
Sep 11 12:25:42.509561 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:42.509510    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fb8846894-krlxl_468a44cb-07a8-4625-bed8-ba64db6c29d7/manager/0.log"
Sep 11 12:25:42.534255 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:42.534201    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fb8846894-krlxl_468a44cb-07a8-4625-bed8-ba64db6c29d7/copy-manifests/0.log"
Sep 11 12:25:43.358812 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:43.358734    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57dff8dc5-hjn5h_047ffb0a-9d8a-42c1-a8ff-83f840a4bf80/kube-auth-proxy/0.log"
Sep 11 12:25:43.506296 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:43.506256    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-5bqf9_c1595fd2-0403-487d-a7ea-3ae02b1309a7/payload-pre-processing/0.log"
Sep 11 12:25:44.004159 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:44.004118    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-866dd874d6-sgpmk_288653f1-8896-4c1c-990a-ecac00f6f0e3/manager/0.log"
Sep 11 12:25:45.567818 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:45.567784    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57dff8dc5-hjn5h_047ffb0a-9d8a-42c1-a8ff-83f840a4bf80/kube-auth-proxy/0.log"
Sep 11 12:25:45.641772 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:45.641737    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-5bqf9_c1595fd2-0403-487d-a7ea-3ae02b1309a7/payload-pre-processing/0.log"
Sep 11 12:25:49.866565 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:49.866521    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-wk5vd_236f4ed7-e91c-49de-8c80-8fb3490942c4/kube-storage-version-migrator-operator/1.log"
Sep 11 12:25:49.868178 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:49.868149    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-wk5vd_236f4ed7-e91c-49de-8c80-8fb3490942c4/kube-storage-version-migrator-operator/0.log"
Sep 11 12:25:51.141372 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:51.141293    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7q5t4_fc1ebadd-d2ed-4de2-8225-21f60ffae756/kube-multus-additional-cni-plugins/0.log"
Sep 11 12:25:51.163847 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:51.163817    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7q5t4_fc1ebadd-d2ed-4de2-8225-21f60ffae756/egress-router-binary-copy/0.log"
Sep 11 12:25:51.185580 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:51.185550    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7q5t4_fc1ebadd-d2ed-4de2-8225-21f60ffae756/cni-plugins/0.log"
Sep 11 12:25:51.208440 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:51.208385    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7q5t4_fc1ebadd-d2ed-4de2-8225-21f60ffae756/bond-cni-plugin/0.log"
Sep 11 12:25:51.230320 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:51.230290    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7q5t4_fc1ebadd-d2ed-4de2-8225-21f60ffae756/routeoverride-cni/0.log"
Sep 11 12:25:51.255573 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:51.255536    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7q5t4_fc1ebadd-d2ed-4de2-8225-21f60ffae756/whereabouts-cni-bincopy/0.log"
Sep 11 12:25:51.279506 ip-10-0-143-220 kubenswrapper[2569]: I0911 12:25:51.279475    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7q5t4_fc1ebadd-d2ed-4de2-8225-21f60ffae756/whereabouts-cni/0.log"