Sep 16 12:00:38.864261 ip-10-0-140-71 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 12:00:39.241792 ip-10-0-140-71 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:00:39.241792 ip-10-0-140-71 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 12:00:39.241792 ip-10-0-140-71 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:00:39.241792 ip-10-0-140-71 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 12:00:39.241792 ip-10-0-140-71 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:00:39.244049 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.243936    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 12:00:39.252009 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.251962    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:00:39.252009 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252005    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:00:39.252009 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252010    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:00:39.252009 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252014    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:00:39.252009 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252016    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:00:39.252009 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252020    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252023    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252026    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252028    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252032    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252034    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252036    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252039    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252042    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252044    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252047    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252049    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252052    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252055    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252057    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252060    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252064    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252067    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252071    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:00:39.252247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252079    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252082    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252084    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252087    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252090    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252092    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252095    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252097    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252100    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252102    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252105    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252107    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252110    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252112    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252115    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252118    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252121    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252124    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252126    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252129    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:00:39.252738 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252131    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252134    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252136    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252139    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252141    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252144    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252146    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252149    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252151    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252154    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252156    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252159    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252162    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252164    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252168    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252170    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252173    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252176    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252178    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252181    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:00:39.253264 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252183    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252185    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252188    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252191    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252193    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252196    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252200    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252202    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252205    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252208    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252211    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252213    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252216    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252218    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252222    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252226    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252229    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252232    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252235    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:00:39.253749 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252238    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252242    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252244    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252247    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252251    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252254    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252257    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252260    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252263    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252265    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252268    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252271    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252273    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252276    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252279    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.252282    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253024    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253034    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253038    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:00:39.254260 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253042    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253045    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253049    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253052    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253055    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253058    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253060    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253063    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253066    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253068    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253071    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253073    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253076    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253078    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253081    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253084    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253087    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253089    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253092    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253094    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:00:39.254716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253097    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253099    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253102    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253104    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253107    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253109    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253111    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253114    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253117    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253119    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253123    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253125    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253129    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253131    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253134    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253136    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253139    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253141    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253144    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253168    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:00:39.255246 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253172    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253175    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253178    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253181    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253184    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253187    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253189    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253194    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253197    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253200    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253202    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253205    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253208    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253211    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253215    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253218    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253221    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253224    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253227    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253230    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:00:39.255739 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253233    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253237    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253239    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253242    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253244    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253248    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253252    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253255    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253257    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253260    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253262    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253265    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253268    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253270    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253273    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253276    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253279    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253281    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253284    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:00:39.256267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253286    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253289    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253292    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253295    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253297    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253300    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253302    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253305    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253307    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253312    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253314    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253317    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253319    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253322    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253324    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253328    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253330    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253719    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253724    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:00:39.256747 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253727    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253730    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253733    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253736    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253740    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253744    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253747    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253750    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253754    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253757    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253760    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253763    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253766    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253768    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253772    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253774    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253777    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253780    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253782    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:00:39.257239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253785    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253787    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253790    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253794    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253796    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253799    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253802    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253805    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253807    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253810    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253813    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253815    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253818    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253821    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253825    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253828    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253831    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253833    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253836    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:00:39.257716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253839    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253842    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253845    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253848    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253850    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253853    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253856    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253858    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253861    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253864    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253866    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253869    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253871    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253874    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253877    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253879    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253882    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253885    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253887    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253890    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:00:39.258202 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253893    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253895    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253898    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253901    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253903    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253906    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253909    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253912    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253914    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253917    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253919    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253922    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253924    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253927    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253929    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253932    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253937    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253940    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253943    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:00:39.258683 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253946    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253949    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253951    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253955    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253958    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253960    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253963    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253965    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253985    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253988    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253991    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253993    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253996    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.253999    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254005    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254008    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254011    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254014    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254017    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254019    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254099    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 12:00:39.259226 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254136    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254144    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254148    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254152    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254156    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254160    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254165    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254168    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254171    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254175    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254178    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254181    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254183    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254186    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254189    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254193    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254196    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254200    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254203    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254206    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254209    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254213    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254217    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 12:00:39.259736 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254220    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254223    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254228    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254231    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254234    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254236    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254239    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254243    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254247    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254250    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254253    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254256    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254258    2582 flags.go:64] FLAG: --enable-server="true"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254261    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254266    2582 flags.go:64] FLAG: --event-burst="100"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254269    2582 flags.go:64] FLAG: --event-qps="50"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254272    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254275    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254278    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254282    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254285    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254287    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254291    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254293    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254296    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 12:00:39.260322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254299    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254302    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254305    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254309    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254312    2582 flags.go:64] FLAG: --feature-gates=""
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254316    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254319    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254322    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254325    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254329    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254332    2582 flags.go:64] FLAG: --help="false"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254335    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254339    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254342    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254345    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254348    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254351    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254354    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254357    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254360    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254362    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254365    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254368    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254371    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 16 12:00:39.260925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254374    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254376    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254379    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254382    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254385    2582 flags.go:64] FLAG: --lock-file=""
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254387    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254390    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254393    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254398    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254401    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254404    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254408    2582 flags.go:64] FLAG: --logging-format="text"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254411    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254414    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254417    2582 flags.go:64] FLAG: --manifest-url=""
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254420    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254425    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254429    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254434    2582 flags.go:64] FLAG: --max-pods="110"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254437    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254440    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254442    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254445    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254448    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254451    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 12:00:39.261698 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254454    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254466    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254469    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254472    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254475    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254477    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254482    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254485    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254488    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254491    2582 flags.go:64] FLAG: --port="10250"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254494    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254496    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0eea082de2a6f4ff3"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254499    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254502    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254505    2582 flags.go:64] FLAG: --register-node="true"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254508    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254512    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254515    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254520    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254522    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254526    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254529    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254531    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254534    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254537    2582 flags.go:64] FLAG: --runonce="false"
Sep 16 12:00:39.262367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254542    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254545    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254548    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254550    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254554    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254557    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254560    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254563    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254566    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254569    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254572    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254574    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254577    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254580    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254583    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254589    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254592    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254594    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254599    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254602    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254604    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254607    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254610    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254613    2582 flags.go:64] FLAG: --v="2"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254621    2582 flags.go:64] FLAG: --version="false"
Sep 16 12:00:39.263043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254628    2582 flags.go:64] FLAG: --vmodule=""
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254632    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.254635    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254747    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254751    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254754    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254756    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254761    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254763    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254766    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254769    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254772    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254774    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254777    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254779    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254782    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254784    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254787    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254789    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254792    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:00:39.263657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254794    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254797    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254799    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254802    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254804    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254807    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254809    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254812    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254815    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254817    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254819    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254822    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254826    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254828    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254831    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254834    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254837    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254839    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254842    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:00:39.264159 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254847    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254850    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254854    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254856    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254859    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254862    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254864    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254868    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254871    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254874    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254877    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254879    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254882    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254884    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254887    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254889    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254892    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254895    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254897    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254900    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:00:39.264633 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254902    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254905    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254907    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254910    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254913    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254917    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254919    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254922    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254925    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254927    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254930    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254932    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254937    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254941    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254944    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254946    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254949    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254952    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254954    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:00:39.265190 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254957    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254959    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254962    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254964    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254982    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254987    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254991    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.254996    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.255000    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.255002    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.255005    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.255008    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.255010    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.255013    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.255015    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.255018    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.255020    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.255023    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.255027    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.255030    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:00:39.265657 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.255032    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:00:39.266164 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.255036    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:00:39.266164 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.255039    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:00:39.266164 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.255042    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:00:39.266164 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.256143    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:00:39.266164 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.263608    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 12:00:39.266164 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.263625    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 12:00:39.266164 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.264323    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 12:00:39.266164 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.264330    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 12:00:39.266164 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264394    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:00:39.266164 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264399    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:00:39.266164 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264402    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:00:39.266164 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264405    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:00:39.266164 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264408    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:00:39.266164 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264411    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264414    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264417    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264419    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264422    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264426    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264430    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264433    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264436    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264439    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264442    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264444    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264447    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264449    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264452    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264455    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264457    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264460    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264462    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264465    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:00:39.266562 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264468    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264470    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264473    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264475    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264479    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264482    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264485    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264488    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264490    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264493    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264496    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264498    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264501    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264503    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264506    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264510    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264514    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264517    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264519    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264522    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:00:39.267066 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264525    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264527    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264530    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264532    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264535    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264538    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264540    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264543    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264545    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264548    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264550    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264553    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264556    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264558    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264561    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264563    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264566    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264570    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264573    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264575    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:00:39.267570 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264578    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264580    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264583    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264585    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264588    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264591    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264593    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264597    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264600    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264603    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264606    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264608    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264610    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264613    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264615    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264618    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264620    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264623    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264625    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:00:39.268069 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264628    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:00:39.268543 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264630    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:00:39.268543 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264633    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:00:39.268543 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264635    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:00:39.268543 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264638    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:00:39.268543 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264642    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:00:39.268543 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264645    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:00:39.268543 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264648    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:00:39.268543 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264650    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:00:39.268543 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264653    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:00:39.268543 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264656    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:00:39.268543 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264659    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:00:39.268543 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264662    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:00:39.268543 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264664    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:00:39.268543 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264667    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:00:39.268543 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.264672    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264824    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264829    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264832    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264834    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264838    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264842    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264845    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264847    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264850    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264852    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264856    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264859    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264862    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264865    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264868    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264870    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264874    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264876    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:00:39.268924 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264879    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264881    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264884    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264886    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264889    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264892    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264895    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264898    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264900    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264903    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264911    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264914    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264917    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264919    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264922    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264925    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264927    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264930    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264933    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:00:39.269414 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264935    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264937    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264940    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264950    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264954    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264957    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264959    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264962    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264964    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264983    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.264989    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265005    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265008    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265011    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265014    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265017    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265020    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265023    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265026    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265028    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:00:39.269871 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265031    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265034    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265037    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265040    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265049    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265052    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265054    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265057    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265059    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265062    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265065    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265067    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265070    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265072    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265074    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265077    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265079    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265082    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265084    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265087    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:00:39.270391 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265089    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265091    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265094    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265097    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265099    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265102    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265104    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265107    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265109    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265112    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265114    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265117    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265119    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265122    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265125    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265127    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265130    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265138    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265140    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:00:39.270869 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265143    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:00:39.271336 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265146    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:00:39.271336 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:39.265148    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:00:39.271336 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.265152    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:00:39.271336 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.265271    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 12:00:39.271336 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.267386    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 12:00:39.271336 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.268165    2582 server.go:1019] "Starting client certificate rotation"
Sep 16 12:00:39.271336 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.268265    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:00:39.271336 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.268301    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:00:39.271336 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.270465    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:00:39.275156 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.275037    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:00:39.282617 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.282599    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 16 12:00:39.285375 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.285361    2582 log.go:25] "Validated CRI v1 image API"
Sep 16 12:00:39.286232 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.286219    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 12:00:39.291066 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.291038    2582 fs.go:135] Filesystem UUIDs: map[27e389d8-cd94-4652-805c-cffb6f2fb04e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9c713515-eed2-46a4-83c9-a625721a8da9:/dev/nvme0n1p4]
Sep 16 12:00:39.291143 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.291062    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 12:00:39.296786 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.296677    2582 manager.go:216] Machine: {Timestamp:2026-09-16 12:00:39.295538873 +0000 UTC m=+0.320840949 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100423 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26a5e995aaee6515863068157d509e SystemUUID:ec26a5e9-95aa-ee65-1586-3068157d509e BootID:418db7ab-a239-4a74-9345-31a939b12a39 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d9:3f:f9:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d9:3f:f9:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:84:99:a9:20:4c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 12:00:39.296786 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.296779    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 12:00:39.296964 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.296857    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 12:00:39.299276 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.299249    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 12:00:39.299428 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.299281    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-71.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 12:00:39.300214 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.300204    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 12:00:39.300252 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.300218    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 12:00:39.300252 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.300229    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:00:39.300815 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.300805    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:00:39.301163 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.301148    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:00:39.301628 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.301617    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 12:00:39.305386 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.305374    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:00:39.305514 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.305504    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 12:00:39.305550 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.305535    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 12:00:39.305550 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.305545    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 12:00:39.305601 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.305562    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 12:00:39.305601 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.305571    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 16 12:00:39.305601 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.305580    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 12:00:39.306064 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.306048    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:00:39.306131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.306069    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:00:39.308664 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.308650    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 12:00:39.311017 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.311000    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 12:00:39.311092 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.311039    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 12:00:39.314187 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.314171    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 12:00:39.314264 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.314194    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 12:00:39.314264 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.314203    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 12:00:39.314264 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.314211    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 12:00:39.314264 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.314224    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 12:00:39.314264 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.314241    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 12:00:39.314264 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.314261    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 12:00:39.314513 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.314271    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 12:00:39.314513 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.314283    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 12:00:39.314513 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.314318    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 12:00:39.314513 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.314344    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 12:00:39.314513 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.314353    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 12:00:39.315156 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.315129    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 12:00:39.315156 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.315152    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 12:00:39.318167 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.318137    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-71.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 12:00:39.318167 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.318143    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 12:00:39.318302 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.318176    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 12:00:39.319273 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.319249    2582 server.go:1268] "Started kubelet"
Sep 16 12:00:39.319352 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.319321    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 12:00:39.319442 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.319416    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 12:00:39.319477 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.319469    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 12:00:39.320261 ip-10-0-140-71 systemd[1]: Started Kubernetes Kubelet.
Sep 16 12:00:39.321234 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.321174    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 12:00:39.321276 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.321180    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 12:00:39.323429 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.323391    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2s4fg"
Sep 16 12:00:39.326582 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.326554    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 12:00:39.328742 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.327555    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-71.ec2.internal.18d5cb5fcfdb2e7e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-71.ec2.internal,UID:ip-10-0-140-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-71.ec2.internal,},FirstTimestamp:2026-09-16 12:00:39.319219838 +0000 UTC m=+0.344521915,LastTimestamp:2026-09-16 12:00:39.319219838 +0000 UTC m=+0.344521915,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-71.ec2.internal,}"
Sep 16 12:00:39.329092 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.329068    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2s4fg"
Sep 16 12:00:39.329612 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.329577    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 12:00:39.329916 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.329902    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 12:00:39.330139 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.330106    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
Sep 16 12:00:39.330563 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.330544    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 12:00:39.330563 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.330548    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 12:00:39.330691 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.330574    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 12:00:39.330734 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.330706    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 12:00:39.330734 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.330715    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 12:00:39.331684 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.331654    2582 factory.go:153] Registering CRI-O factory
Sep 16 12:00:39.331684 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.331680    2582 factory.go:223] Registration of the crio container factory successfully
Sep 16 12:00:39.331807 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.331752    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 12:00:39.331807 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.331764    2582 factory.go:55] Registering systemd factory
Sep 16 12:00:39.331807 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.331774    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 16 12:00:39.331807 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.331799    2582 factory.go:103] Registering Raw factory
Sep 16 12:00:39.332011 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.331817    2582 manager.go:1196] Started watching for new ooms in manager
Sep 16 12:00:39.332565 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.332551    2582 manager.go:349] Starting recovery of all containers
Sep 16 12:00:39.333877 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.333853    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 12:00:39.334152 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.334100    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 12:00:39.344196 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.344168    2582 manager.go:354] Recovery completed
Sep 16 12:00:39.346795 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.346757    2582 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 16 12:00:39.349849 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.349835    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:00:39.352605 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.352586    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:00:39.352668 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.352615    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:00:39.352668 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.352625    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:00:39.353278 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.353263    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 12:00:39.353278 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.353276    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 12:00:39.353278 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.353293    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:00:39.353278 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.353300    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:00:39.355255 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.355240    2582 policy_none.go:49] "None policy: Start"
Sep 16 12:00:39.355255 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.355257    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 12:00:39.355370 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.355267    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 12:00:39.355667 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.355645    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:00:39.355708 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.355673    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:00:39.355708 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.355686    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:00:39.357296 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.357285    2582 policy_none.go:47] "Start"
Sep 16 12:00:39.412930 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.398488    2582 manager.go:337] "Starting Device Plugin manager"
Sep 16 12:00:39.412930 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.398538    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 12:00:39.412930 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.398551    2582 server.go:87] "Starting device plugin registration server"
Sep 16 12:00:39.412930 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.398873    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 12:00:39.412930 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.398888    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 12:00:39.412930 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.399007    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 12:00:39.412930 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.399107    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 12:00:39.412930 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.399119    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 12:00:39.412930 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.399625    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 12:00:39.412930 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.399664    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-71.ec2.internal\" not found"
Sep 16 12:00:39.429343 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.429311    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 12:00:39.430645 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.430625    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 12:00:39.430764 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.430680    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 12:00:39.430764 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.430709    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 12:00:39.430764 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.430752    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 12:00:39.433841 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.433818    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:00:39.499251 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.499026    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:00:39.500198 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.500177    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:00:39.500296 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.500218    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:00:39.500296 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.500233    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:00:39.500296 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.500264    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.508194 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.508179    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.508262 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.508201    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-71.ec2.internal\": node \"ip-10-0-140-71.ec2.internal\" not found"
Sep 16 12:00:39.522954 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.522927    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
Sep 16 12:00:39.531017 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.530993    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal"]
Sep 16 12:00:39.531084 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.531075    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:00:39.533202 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.533187    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:00:39.533273 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.533219    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:00:39.533273 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.533233    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:00:39.535516 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.535502    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:00:39.535654 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.535639    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.535711 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.535673    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:00:39.536341 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.536317    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:00:39.536434 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.536352    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:00:39.536434 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.536363    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:00:39.536434 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.536321    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:00:39.536434 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.536430    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:00:39.536565 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.536443    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:00:39.538552 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.538534    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.538629 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.538565    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:00:39.539578 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.539561    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:00:39.539688 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.539589    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:00:39.539688 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.539601    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:00:39.555550 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.555524    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-71.ec2.internal\" not found" node="ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.559499 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.559482    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-71.ec2.internal\" not found" node="ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.623908 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.623877    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
Sep 16 12:00:39.724224 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.724186    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
Sep 16 12:00:39.731575 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.731542    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/623b25dd964a2b389941b772b9ff3ebf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal\" (UID: \"623b25dd964a2b389941b772b9ff3ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.731670 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.731582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/623b25dd964a2b389941b772b9ff3ebf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal\" (UID: \"623b25dd964a2b389941b772b9ff3ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.731670 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.731612    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/daddab1c0f05fb32f4f8c0ebec22f11c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-71.ec2.internal\" (UID: \"daddab1c0f05fb32f4f8c0ebec22f11c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.825015 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.824921    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
Sep 16 12:00:39.832259 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.832236    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/daddab1c0f05fb32f4f8c0ebec22f11c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-71.ec2.internal\" (UID: \"daddab1c0f05fb32f4f8c0ebec22f11c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.832324 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.832268    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/623b25dd964a2b389941b772b9ff3ebf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal\" (UID: \"623b25dd964a2b389941b772b9ff3ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.832324 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.832291    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/623b25dd964a2b389941b772b9ff3ebf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal\" (UID: \"623b25dd964a2b389941b772b9ff3ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.832422 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.832347    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/daddab1c0f05fb32f4f8c0ebec22f11c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-71.ec2.internal\" (UID: \"daddab1c0f05fb32f4f8c0ebec22f11c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.832422 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.832394    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/623b25dd964a2b389941b772b9ff3ebf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal\" (UID: \"623b25dd964a2b389941b772b9ff3ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.832422 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.832419    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/623b25dd964a2b389941b772b9ff3ebf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal\" (UID: \"623b25dd964a2b389941b772b9ff3ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.858396 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.858360    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.861912 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:39.861885    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal"
Sep 16 12:00:39.925986 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:39.925944    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
Sep 16 12:00:40.026501 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:40.026469    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
Sep 16 12:00:40.127121 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:40.127015    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
Sep 16 12:00:40.227601 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:40.227555    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
Sep 16 12:00:40.268032 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:40.268004    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 12:00:40.268463 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:40.268174    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:00:40.328854 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:40.328824    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
Sep 16 12:00:40.329297 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:40.329277    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:00:40.335490 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:40.335449    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 11:55:41 +0000 UTC" deadline="2028-05-31 23:04:31.146205622 +0000 UTC"
Sep 16 12:00:40.335589 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:40.335490    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14963h3m50.810721294s"
Sep 16 12:00:40.414276 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:40.414241    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623b25dd964a2b389941b772b9ff3ebf.slice/crio-ac49ee4abdb1eaa293dfc7cb25e420e036a0af45b369f9be562ca10d6325e948 WatchSource:0}: Error finding container ac49ee4abdb1eaa293dfc7cb25e420e036a0af45b369f9be562ca10d6325e948: Status 404 returned error can't find the container with id ac49ee4abdb1eaa293dfc7cb25e420e036a0af45b369f9be562ca10d6325e948
Sep 16 12:00:40.414530 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:40.414512    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaddab1c0f05fb32f4f8c0ebec22f11c.slice/crio-543655c489fd6ef6ab863ff30d12481a4308ae9f0a62365b0f190f6ca7675ea5 WatchSource:0}: Error finding container 543655c489fd6ef6ab863ff30d12481a4308ae9f0a62365b0f190f6ca7675ea5: Status 404 returned error can't find the container with id 543655c489fd6ef6ab863ff30d12481a4308ae9f0a62365b0f190f6ca7675ea5
Sep 16 12:00:40.419877 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:40.419854    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:00:40.429399 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:40.429378    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
Sep 16 12:00:40.434257 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:40.434218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal" event={"ID":"daddab1c0f05fb32f4f8c0ebec22f11c","Type":"ContainerStarted","Data":"543655c489fd6ef6ab863ff30d12481a4308ae9f0a62365b0f190f6ca7675ea5"}
Sep 16 12:00:40.435409 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:40.435385    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal" event={"ID":"623b25dd964a2b389941b772b9ff3ebf","Type":"ContainerStarted","Data":"ac49ee4abdb1eaa293dfc7cb25e420e036a0af45b369f9be562ca10d6325e948"}
Sep 16 12:00:40.529720 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:40.529676    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
Sep 16 12:00:40.630143 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:40.630101    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
Sep 16 12:00:40.710757 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:40.710689    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:00:40.730829 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:40.730782    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
Sep 16 12:00:40.741432 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:40.741410    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:00:40.742826 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:40.742810    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal"
Sep 16 12:00:40.749444 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:40.749417    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:00:42.682104 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:42.682080    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:00:43.081459 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.081429    2582 apiserver.go:52] "Watching apiserver"
Sep 16 12:00:43.090596 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.090570    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 12:00:43.091028 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.091002    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-8drdm","kube-system/konnectivity-agent-z7wfg","kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq","openshift-cluster-node-tuning-operator/tuned-54c4z","openshift-dns/node-resolver-4hzq6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal","openshift-multus/multus-5smzb","openshift-image-registry/node-ca-7q8hz","openshift-multus/multus-additional-cni-plugins-w69gx","openshift-multus/network-metrics-daemon-bflx7","openshift-network-diagnostics/network-check-target-2ps6l","openshift-network-operator/iptables-alerter-2kpnf","openshift-ovn-kubernetes/ovnkube-node-mcx6d"]
Sep 16 12:00:43.093543 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.093521    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.095750 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.095701    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2kpnf"
Sep 16 12:00:43.096398 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.096379    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 12:00:43.096503 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.096443    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4l8gt\""
Sep 16 12:00:43.096716 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.096687    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 12:00:43.096833 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.096728    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 12:00:43.096833 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.096768    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 12:00:43.096833 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.096807    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 12:00:43.098159 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.098140    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 12:00:43.098299 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.098280    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.098882 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.098861    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:00:43.098980 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.098931    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-45qxq\""
Sep 16 12:00:43.099040 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.098865    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 12:00:43.100352 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.100333    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 12:00:43.100710 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.100693    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.101654 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.101637    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 12:00:43.101760 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.101705    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 12:00:43.101760 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.101717    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 12:00:43.101869 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.101798    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 12:00:43.101918 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.101878    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 12:00:43.101959 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.101925    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fcqrk\""
Sep 16 12:00:43.102886 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.102792    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 12:00:43.103068 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.103049    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 12:00:43.103129 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.103063    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 12:00:43.103330 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.103312    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gb646\""
Sep 16 12:00:43.103393 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.103362    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.105295 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.105275    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 12:00:43.105434 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.105418    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4hzq6"
Sep 16 12:00:43.105664 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.105646    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-klnsl\""
Sep 16 12:00:43.105881 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.105863    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:00:43.105998 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.105981    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 12:00:43.107762 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.107728    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 12:00:43.107975 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.107751    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7q8hz"
Sep 16 12:00:43.109334 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.108792    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jgr2j\""
Sep 16 12:00:43.109334 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.108814    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 12:00:43.110247 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.110227    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 12:00:43.110760 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.110357    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.110760 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.110402    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 12:00:43.110760 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.110635    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 12:00:43.110946 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.110904    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vjzqc\""
Sep 16 12:00:43.112848 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.112830    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z7wfg"
Sep 16 12:00:43.113953 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.113936    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 12:00:43.114094 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114036    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-run-netns\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.114094 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114087    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hcwlc\""
Sep 16 12:00:43.114217 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114072    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.114217 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9fnn\" (UniqueName: \"kubernetes.io/projected/e5308201-4e49-461a-be86-4c661c0d2783-kube-api-access-x9fnn\") pod \"node-resolver-4hzq6\" (UID: \"e5308201-4e49-461a-be86-4c661c0d2783\") " pod="openshift-dns/node-resolver-4hzq6"
Sep 16 12:00:43.114217 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114183    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-tuned\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.114217 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114207    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55b45601-95e3-4427-af82-2358088e7e8a-host\") pod \"node-ca-7q8hz\" (UID: \"55b45601-95e3-4427-af82-2358088e7e8a\") " pod="openshift-image-registry/node-ca-7q8hz"
Sep 16 12:00:43.114427 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114233    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-run-ovn-kubernetes\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.114427 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114257    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-sysconfig\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.114427 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114280    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-kubernetes\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.114427 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m597l\" (UniqueName: \"kubernetes.io/projected/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-kube-api-access-m597l\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.114427 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114333    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-sys-fs\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.114427 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114356    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25085fb2-c567-4b20-bd8d-6241d7cce1cf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.114427 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv5br\" (UniqueName: \"kubernetes.io/projected/55b45601-95e3-4427-af82-2358088e7e8a-kube-api-access-mv5br\") pod \"node-ca-7q8hz\" (UID: \"55b45601-95e3-4427-af82-2358088e7e8a\") " pod="openshift-image-registry/node-ca-7q8hz"
Sep 16 12:00:43.114427 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114415    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/030e2725-fa96-4218-9026-a4a319a47ab4-ovnkube-script-lib\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.114792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114437    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-sysctl-conf\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.114792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114460    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/55b45601-95e3-4427-af82-2358088e7e8a-serviceca\") pod \"node-ca-7q8hz\" (UID: \"55b45601-95e3-4427-af82-2358088e7e8a\") " pod="openshift-image-registry/node-ca-7q8hz"
Sep 16 12:00:43.114792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-node-log\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.114792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114518    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25085fb2-c567-4b20-bd8d-6241d7cce1cf-cni-binary-copy\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.114792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114544    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25085fb2-c567-4b20-bd8d-6241d7cce1cf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.114792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114568    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-var-lib-kubelet\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.114792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114613    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-var-lib-openvswitch\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.114792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114647    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-socket-dir\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.114792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/020357fa-c14e-4e1d-babf-bb7910790e41-host-slash\") pod \"iptables-alerter-2kpnf\" (UID: \"020357fa-c14e-4e1d-babf-bb7910790e41\") " pod="openshift-network-operator/iptables-alerter-2kpnf"
Sep 16 12:00:43.114792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25085fb2-c567-4b20-bd8d-6241d7cce1cf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.114792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114778    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-sysctl-d\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.115212 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114807    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-etc-openvswitch\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.115212 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr7kd\" (UniqueName: \"kubernetes.io/projected/030e2725-fa96-4218-9026-a4a319a47ab4-kube-api-access-mr7kd\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.115212 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114868    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-etc-selinux\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.115212 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114913    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-cni-bin\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.115212 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114951    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/030e2725-fa96-4218-9026-a4a319a47ab4-ovn-node-metrics-cert\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.115212 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.114978    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-modprobe-d\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.115212 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115003    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e5308201-4e49-461a-be86-4c661c0d2783-hosts-file\") pod \"node-resolver-4hzq6\" (UID: \"e5308201-4e49-461a-be86-4c661c0d2783\") " pod="openshift-dns/node-resolver-4hzq6"
Sep 16 12:00:43.115212 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115028    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/020357fa-c14e-4e1d-babf-bb7910790e41-iptables-alerter-script\") pod \"iptables-alerter-2kpnf\" (UID: \"020357fa-c14e-4e1d-babf-bb7910790e41\") " pod="openshift-network-operator/iptables-alerter-2kpnf"
Sep 16 12:00:43.115212 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115049    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-kubelet\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.115212 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115077    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-slash\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.115212 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115103    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25085fb2-c567-4b20-bd8d-6241d7cce1cf-system-cni-dir\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.115212 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115118    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5308201-4e49-461a-be86-4c661c0d2783-tmp-dir\") pod \"node-resolver-4hzq6\" (UID: \"e5308201-4e49-461a-be86-4c661c0d2783\") " pod="openshift-dns/node-resolver-4hzq6"
Sep 16 12:00:43.115212 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115132    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/030e2725-fa96-4218-9026-a4a319a47ab4-ovnkube-config\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.115212 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115155    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-systemd\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.115212 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-run-openvswitch\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.115212 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115200    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-log-socket\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.116088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115227    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-registration-dir\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.116088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115257    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-run\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.116088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115276    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-lib-modules\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.116088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115292    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-run-ovn\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.116088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115327    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.116088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlpvx\" (UniqueName: \"kubernetes.io/projected/25085fb2-c567-4b20-bd8d-6241d7cce1cf-kube-api-access-hlpvx\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.116088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115372    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-sys\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.116088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115394    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-run-systemd\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.116088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115431    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/030e2725-fa96-4218-9026-a4a319a47ab4-env-overrides\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.116088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115506    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-systemd-units\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.116088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115537    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-device-dir\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.116088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115564    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbf8h\" (UniqueName: \"kubernetes.io/projected/5ff1c306-c904-4cea-8c0b-e0083708fe0f-kube-api-access-kbf8h\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.116088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115621    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-host\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.116088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115646    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-tmp\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.116088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25085fb2-c567-4b20-bd8d-6241d7cce1cf-cnibin\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.116088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115697    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ldsg\" (UniqueName: \"kubernetes.io/projected/020357fa-c14e-4e1d-babf-bb7910790e41-kube-api-access-6ldsg\") pod \"iptables-alerter-2kpnf\" (UID: \"020357fa-c14e-4e1d-babf-bb7910790e41\") " pod="openshift-network-operator/iptables-alerter-2kpnf"
Sep 16 12:00:43.116700 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115758    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-cni-netd\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.116700 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115785    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 12:00:43.116700 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115817    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9hkhs\""
Sep 16 12:00:43.116700 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.115828    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 12:00:43.116700 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.116181    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:43.116700 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.116254    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:00:43.118759 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.118723    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:43.118843 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.118803    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:00:43.120562 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.120542    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:00:43.121222 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.121204    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:00:43.121317 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.121275    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ps6l" podUID="becc876e-d896-4811-9ee1-0be40fe69dda"
Sep 16 12:00:43.138607 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.138577    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6bcps"
Sep 16 12:00:43.145854 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.145831    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6bcps"
Sep 16 12:00:43.206359 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.206333    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 12:00:43.216772 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.216721    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-tuned\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.216772 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.216777    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55b45601-95e3-4427-af82-2358088e7e8a-host\") pod \"node-ca-7q8hz\" (UID: \"55b45601-95e3-4427-af82-2358088e7e8a\") " pod="openshift-image-registry/node-ca-7q8hz"
Sep 16 12:00:43.217002 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.216795    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-run-ovn-kubernetes\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.217002 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.216812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-multus-socket-dir-parent\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.217002 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.216840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-multus-conf-dir\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.217002 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.216854    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-sysconfig\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.217002 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.216882    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-kubernetes\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.217002 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.216892    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-run-ovn-kubernetes\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.217002 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.216907    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m597l\" (UniqueName: \"kubernetes.io/projected/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-kube-api-access-m597l\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.217002 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.216929    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55b45601-95e3-4427-af82-2358088e7e8a-host\") pod \"node-ca-7q8hz\" (UID: \"55b45601-95e3-4427-af82-2358088e7e8a\") " pod="openshift-image-registry/node-ca-7q8hz"
Sep 16 12:00:43.217333 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.217228    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-sysconfig\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.217333 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.217288    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-kubernetes\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.217489 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.217348    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 12:00:43.217489 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.217426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-sys-fs\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.217489 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.217479    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25085fb2-c567-4b20-bd8d-6241d7cce1cf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.217639 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.217507    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv5br\" (UniqueName: \"kubernetes.io/projected/55b45601-95e3-4427-af82-2358088e7e8a-kube-api-access-mv5br\") pod \"node-ca-7q8hz\" (UID: \"55b45601-95e3-4427-af82-2358088e7e8a\") " pod="openshift-image-registry/node-ca-7q8hz"
Sep 16 12:00:43.217639 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.217537    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/030e2725-fa96-4218-9026-a4a319a47ab4-ovnkube-script-lib\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.217639 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.217580    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-sys-fs\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.217639 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.217571    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-system-cni-dir\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.217857 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.217675    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bef09744-222b-4d55-9d87-aca589bd4f13-kubelet-config\") pod \"global-pull-secret-syncer-8drdm\" (UID: \"bef09744-222b-4d55-9d87-aca589bd4f13\") " pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:43.217857 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.217768    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-sysctl-conf\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.217857 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.217803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/55b45601-95e3-4427-af82-2358088e7e8a-serviceca\") pod \"node-ca-7q8hz\" (UID: \"55b45601-95e3-4427-af82-2358088e7e8a\") " pod="openshift-image-registry/node-ca-7q8hz"
Sep 16 12:00:43.217857 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.217840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-multus-cni-dir\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.218044 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.217976    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-run-multus-certs\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.218044 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218021    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret\") pod \"global-pull-secret-syncer-8drdm\" (UID: \"bef09744-222b-4d55-9d87-aca589bd4f13\") " pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:43.218149 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218062    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-node-log\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.218149 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218138    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-node-log\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.218246 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218193    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25085fb2-c567-4b20-bd8d-6241d7cce1cf-cni-binary-copy\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.218246 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218230    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25085fb2-c567-4b20-bd8d-6241d7cce1cf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.218346 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-var-lib-kubelet\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.218396 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218347    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c51bc19e-789f-4167-80b8-ada031cdfc55-agent-certs\") pod \"konnectivity-agent-z7wfg\" (UID: \"c51bc19e-789f-4167-80b8-ada031cdfc55\") " pod="kube-system/konnectivity-agent-z7wfg"
Sep 16 12:00:43.218396 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218365    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/55b45601-95e3-4427-af82-2358088e7e8a-serviceca\") pod \"node-ca-7q8hz\" (UID: \"55b45601-95e3-4427-af82-2358088e7e8a\") " pod="openshift-image-registry/node-ca-7q8hz"
Sep 16 12:00:43.218396 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218384    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-var-lib-openvswitch\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.218538 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-socket-dir\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.218538 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218455    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/020357fa-c14e-4e1d-babf-bb7910790e41-host-slash\") pod \"iptables-alerter-2kpnf\" (UID: \"020357fa-c14e-4e1d-babf-bb7910790e41\") " pod="openshift-network-operator/iptables-alerter-2kpnf"
Sep 16 12:00:43.218538 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218491    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25085fb2-c567-4b20-bd8d-6241d7cce1cf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.218680 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218534    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25085fb2-c567-4b20-bd8d-6241d7cce1cf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.218729 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218688    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-sysctl-d\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.218729 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218723    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-etc-openvswitch\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.218857 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218774    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr7kd\" (UniqueName: \"kubernetes.io/projected/030e2725-fa96-4218-9026-a4a319a47ab4-kube-api-access-mr7kd\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.218857 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-etc-selinux\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.218857 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218822    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-socket-dir\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.218993 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218833    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/020357fa-c14e-4e1d-babf-bb7910790e41-host-slash\") pod \"iptables-alerter-2kpnf\" (UID: \"020357fa-c14e-4e1d-babf-bb7910790e41\") " pod="openshift-network-operator/iptables-alerter-2kpnf"
Sep 16 12:00:43.218993 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-cni-bin\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.218993 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.218947    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-var-lib-openvswitch\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.219181 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219070    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-etc-selinux\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.219233 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219196    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-sysctl-d\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.219286 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219262    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-etc-openvswitch\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.219339 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219326    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-var-lib-kubelet\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.219428 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/030e2725-fa96-4218-9026-a4a319a47ab4-ovn-node-metrics-cert\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.219511 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-modprobe-d\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.219511 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219473    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25085fb2-c567-4b20-bd8d-6241d7cce1cf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.219511 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219478    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e5308201-4e49-461a-be86-4c661c0d2783-hosts-file\") pod \"node-resolver-4hzq6\" (UID: \"e5308201-4e49-461a-be86-4c661c0d2783\") " pod="openshift-dns/node-resolver-4hzq6"
Sep 16 12:00:43.219662 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219535    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e5308201-4e49-461a-be86-4c661c0d2783-hosts-file\") pod \"node-resolver-4hzq6\" (UID: \"e5308201-4e49-461a-be86-4c661c0d2783\") " pod="openshift-dns/node-resolver-4hzq6"
Sep 16 12:00:43.219662 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/020357fa-c14e-4e1d-babf-bb7910790e41-iptables-alerter-script\") pod \"iptables-alerter-2kpnf\" (UID: \"020357fa-c14e-4e1d-babf-bb7910790e41\") " pod="openshift-network-operator/iptables-alerter-2kpnf"
Sep 16 12:00:43.219662 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219625    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-kubelet\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.219839 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219660    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-slash\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.219839 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3fc4ace4-ea76-4d43-8722-093f76f186dd-multus-daemon-config\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.219839 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219696    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25085fb2-c567-4b20-bd8d-6241d7cce1cf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.219839 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219805    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-modprobe-d\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.220027 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219886    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/030e2725-fa96-4218-9026-a4a319a47ab4-ovnkube-script-lib\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.220027 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219951    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25085fb2-c567-4b20-bd8d-6241d7cce1cf-system-cni-dir\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.220027 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.219991    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5308201-4e49-461a-be86-4c661c0d2783-tmp-dir\") pod \"node-resolver-4hzq6\" (UID: \"e5308201-4e49-461a-be86-4c661c0d2783\") " pod="openshift-dns/node-resolver-4hzq6"
Sep 16 12:00:43.220027 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220021    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/030e2725-fa96-4218-9026-a4a319a47ab4-ovnkube-config\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.220215 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220058    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-run-k8s-cni-cncf-io\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.220215 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220073    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-cni-bin\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.220215 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220095    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x5nk\" (UniqueName: \"kubernetes.io/projected/3fc4ace4-ea76-4d43-8722-093f76f186dd-kube-api-access-8x5nk\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.220356 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220276    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-kubelet\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.220356 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-sysctl-conf\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.220679 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220552    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-systemd\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.220679 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220597    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-run-openvswitch\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.220679 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220596    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/020357fa-c14e-4e1d-babf-bb7910790e41-iptables-alerter-script\") pod \"iptables-alerter-2kpnf\" (UID: \"020357fa-c14e-4e1d-babf-bb7910790e41\") " pod="openshift-network-operator/iptables-alerter-2kpnf"
Sep 16 12:00:43.220679 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220659    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-slash\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.220958 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-log-socket\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.220958 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220771    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bef09744-222b-4d55-9d87-aca589bd4f13-dbus\") pod \"global-pull-secret-syncer-8drdm\" (UID: \"bef09744-222b-4d55-9d87-aca589bd4f13\") " pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:43.220958 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220801    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs\") pod \"network-metrics-daemon-bflx7\" (UID: \"9a90e5c3-5fcb-4016-8682-3803c3121620\") " pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:43.220958 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-registration-dir\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.220958 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-run\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.220958 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220862    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25085fb2-c567-4b20-bd8d-6241d7cce1cf-cni-binary-copy\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.220958 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220875    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/030e2725-fa96-4218-9026-a4a319a47ab4-ovnkube-config\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.220958 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220939    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-registration-dir\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.220958 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220949    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25085fb2-c567-4b20-bd8d-6241d7cce1cf-system-cni-dir\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.220958 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220956    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-lib-modules\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.221476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-run-ovn\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.221476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221029    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-var-lib-cni-bin\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.221476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221066    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c51bc19e-789f-4167-80b8-ada031cdfc55-konnectivity-ca\") pod \"konnectivity-agent-z7wfg\" (UID: \"c51bc19e-789f-4167-80b8-ada031cdfc55\") " pod="kube-system/konnectivity-agent-z7wfg"
Sep 16 12:00:43.221476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221148    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzrkw\" (UniqueName: \"kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw\") pod \"network-check-target-2ps6l\" (UID: \"becc876e-d896-4811-9ee1-0be40fe69dda\") " pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:00:43.221476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221187    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.221476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221208    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5308201-4e49-461a-be86-4c661c0d2783-tmp-dir\") pod \"node-resolver-4hzq6\" (UID: \"e5308201-4e49-461a-be86-4c661c0d2783\") " pod="openshift-dns/node-resolver-4hzq6"
Sep 16 12:00:43.221476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlpvx\" (UniqueName: \"kubernetes.io/projected/25085fb2-c567-4b20-bd8d-6241d7cce1cf-kube-api-access-hlpvx\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.221476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-sys\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.221476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-run-systemd\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.221476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221329    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/030e2725-fa96-4218-9026-a4a319a47ab4-env-overrides\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.221476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-run-netns\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.221476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221389    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-tuned\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.221476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221395    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-systemd-units\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.222115 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221512    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-run-ovn\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.222115 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221566    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-systemd-units\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.222115 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221613    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-sys\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.222115 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221657    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-run-systemd\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.222115 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221688    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-device-dir\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.222115 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221807    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-run\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.222115 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221857    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-etc-systemd\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.222115 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221896    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-run-openvswitch\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.222115 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221930    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbf8h\" (UniqueName: \"kubernetes.io/projected/5ff1c306-c904-4cea-8c0b-e0083708fe0f-kube-api-access-kbf8h\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.222115 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221943    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-device-dir\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.222115 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.221978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-host\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.222115 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222021    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-host\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.222115 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222040    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ff1c306-c904-4cea-8c0b-e0083708fe0f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.222115 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222069    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/030e2725-fa96-4218-9026-a4a319a47ab4-env-overrides\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.222115 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222106    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-lib-modules\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.223016 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222236    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-tmp\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.223016 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222301    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3fc4ace4-ea76-4d43-8722-093f76f186dd-cni-binary-copy\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.223016 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25085fb2-c567-4b20-bd8d-6241d7cce1cf-cnibin\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.223016 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ldsg\" (UniqueName: \"kubernetes.io/projected/020357fa-c14e-4e1d-babf-bb7910790e41-kube-api-access-6ldsg\") pod \"iptables-alerter-2kpnf\" (UID: \"020357fa-c14e-4e1d-babf-bb7910790e41\") " pod="openshift-network-operator/iptables-alerter-2kpnf"
Sep 16 12:00:43.223016 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-cni-netd\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.223016 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222462    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-cnibin\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.223016 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-var-lib-cni-multus\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.223016 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-var-lib-kubelet\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.223016 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-hostroot\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.223016 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222613    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-etc-kubernetes\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.223016 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222641    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-run-netns\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.223016 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.223016 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222707    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l96jl\" (UniqueName: \"kubernetes.io/projected/9a90e5c3-5fcb-4016-8682-3803c3121620-kube-api-access-l96jl\") pod \"network-metrics-daemon-bflx7\" (UID: \"9a90e5c3-5fcb-4016-8682-3803c3121620\") " pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:43.223016 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.222762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9fnn\" (UniqueName: \"kubernetes.io/projected/e5308201-4e49-461a-be86-4c661c0d2783-kube-api-access-x9fnn\") pod \"node-resolver-4hzq6\" (UID: \"e5308201-4e49-461a-be86-4c661c0d2783\") " pod="openshift-dns/node-resolver-4hzq6"
Sep 16 12:00:43.223805 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.223286    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25085fb2-c567-4b20-bd8d-6241d7cce1cf-cnibin\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.223805 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.223322    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-cni-netd\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.223805 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.223372    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-run-netns\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.223805 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.223408    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.225023 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.220989    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/030e2725-fa96-4218-9026-a4a319a47ab4-log-socket\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.225110 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.225068    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-tmp\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.226278 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.226257    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/030e2725-fa96-4218-9026-a4a319a47ab4-ovn-node-metrics-cert\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.231369 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.231349    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv5br\" (UniqueName: \"kubernetes.io/projected/55b45601-95e3-4427-af82-2358088e7e8a-kube-api-access-mv5br\") pod \"node-ca-7q8hz\" (UID: \"55b45601-95e3-4427-af82-2358088e7e8a\") " pod="openshift-image-registry/node-ca-7q8hz"
Sep 16 12:00:43.231720 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.231681    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m597l\" (UniqueName: \"kubernetes.io/projected/c0064df1-27fe-4e46-bc7a-4ef75b8d9892-kube-api-access-m597l\") pod \"tuned-54c4z\" (UID: \"c0064df1-27fe-4e46-bc7a-4ef75b8d9892\") " pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.231834 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.231722    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr7kd\" (UniqueName: \"kubernetes.io/projected/030e2725-fa96-4218-9026-a4a319a47ab4-kube-api-access-mr7kd\") pod \"ovnkube-node-mcx6d\" (UID: \"030e2725-fa96-4218-9026-a4a319a47ab4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.232183 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.232156    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlpvx\" (UniqueName: \"kubernetes.io/projected/25085fb2-c567-4b20-bd8d-6241d7cce1cf-kube-api-access-hlpvx\") pod \"multus-additional-cni-plugins-w69gx\" (UID: \"25085fb2-c567-4b20-bd8d-6241d7cce1cf\") " pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.233380 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.233355    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ldsg\" (UniqueName: \"kubernetes.io/projected/020357fa-c14e-4e1d-babf-bb7910790e41-kube-api-access-6ldsg\") pod \"iptables-alerter-2kpnf\" (UID: \"020357fa-c14e-4e1d-babf-bb7910790e41\") " pod="openshift-network-operator/iptables-alerter-2kpnf"
Sep 16 12:00:43.233632 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.233611    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbf8h\" (UniqueName: \"kubernetes.io/projected/5ff1c306-c904-4cea-8c0b-e0083708fe0f-kube-api-access-kbf8h\") pod \"aws-ebs-csi-driver-node-lqpxq\" (UID: \"5ff1c306-c904-4cea-8c0b-e0083708fe0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.234000 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.233979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9fnn\" (UniqueName: \"kubernetes.io/projected/e5308201-4e49-461a-be86-4c661c0d2783-kube-api-access-x9fnn\") pod \"node-resolver-4hzq6\" (UID: \"e5308201-4e49-461a-be86-4c661c0d2783\") " pod="openshift-dns/node-resolver-4hzq6"
Sep 16 12:00:43.317952 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.317919    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:00:43.323565 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323537    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3fc4ace4-ea76-4d43-8722-093f76f186dd-cni-binary-copy\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.323565 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323574    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-cnibin\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.323796 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323599    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-var-lib-cni-multus\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.323796 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323652    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-var-lib-cni-multus\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.323796 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323689    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-cnibin\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.323796 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323752    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-var-lib-kubelet\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.323796 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-hostroot\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323815    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-etc-kubernetes\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323842    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-var-lib-kubelet\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323853    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-etc-kubernetes\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323859    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-hostroot\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l96jl\" (UniqueName: \"kubernetes.io/projected/9a90e5c3-5fcb-4016-8682-3803c3121620-kube-api-access-l96jl\") pod \"network-metrics-daemon-bflx7\" (UID: \"9a90e5c3-5fcb-4016-8682-3803c3121620\") " pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:43.324034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323895    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-multus-socket-dir-parent\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323912    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-multus-conf-dir\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-system-cni-dir\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323948    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bef09744-222b-4d55-9d87-aca589bd4f13-kubelet-config\") pod \"global-pull-secret-syncer-8drdm\" (UID: \"bef09744-222b-4d55-9d87-aca589bd4f13\") " pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:43.324034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-multus-cni-dir\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-multus-socket-dir-parent\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-run-multus-certs\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324013    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret\") pod \"global-pull-secret-syncer-8drdm\" (UID: \"bef09744-222b-4d55-9d87-aca589bd4f13\") " pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:43.324034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.323991    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-multus-conf-dir\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324038    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-system-cni-dir\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324049    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bef09744-222b-4d55-9d87-aca589bd4f13-kubelet-config\") pod \"global-pull-secret-syncer-8drdm\" (UID: \"bef09744-222b-4d55-9d87-aca589bd4f13\") " pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324072    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-run-multus-certs\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c51bc19e-789f-4167-80b8-ada031cdfc55-agent-certs\") pod \"konnectivity-agent-z7wfg\" (UID: \"c51bc19e-789f-4167-80b8-ada031cdfc55\") " pod="kube-system/konnectivity-agent-z7wfg"
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324129    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-multus-cni-dir\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324132    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3fc4ace4-ea76-4d43-8722-093f76f186dd-multus-daemon-config\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.324132    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324171    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-run-k8s-cni-cncf-io\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324195    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5nk\" (UniqueName: \"kubernetes.io/projected/3fc4ace4-ea76-4d43-8722-093f76f186dd-kube-api-access-8x5nk\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.324236    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret podName:bef09744-222b-4d55-9d87-aca589bd4f13 nodeName:}" failed. No retries permitted until 2026-09-16 12:00:43.824211912 +0000 UTC m=+3.074749439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret") pod "global-pull-secret-syncer-8drdm" (UID: "bef09744-222b-4d55-9d87-aca589bd4f13") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bef09744-222b-4d55-9d87-aca589bd4f13-dbus\") pod \"global-pull-secret-syncer-8drdm\" (UID: \"bef09744-222b-4d55-9d87-aca589bd4f13\") " pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324262    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-run-k8s-cni-cncf-io\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324282    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3fc4ace4-ea76-4d43-8722-093f76f186dd-cni-binary-copy\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324284    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs\") pod \"network-metrics-daemon-bflx7\" (UID: \"9a90e5c3-5fcb-4016-8682-3803c3121620\") " pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324330    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bef09744-222b-4d55-9d87-aca589bd4f13-dbus\") pod \"global-pull-secret-syncer-8drdm\" (UID: \"bef09744-222b-4d55-9d87-aca589bd4f13\") " pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324335    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-var-lib-cni-bin\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.324349    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:00:43.324713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324365    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c51bc19e-789f-4167-80b8-ada031cdfc55-konnectivity-ca\") pod \"konnectivity-agent-z7wfg\" (UID: \"c51bc19e-789f-4167-80b8-ada031cdfc55\") " pod="kube-system/konnectivity-agent-z7wfg"
Sep 16 12:00:43.325405 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324372    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-var-lib-cni-bin\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.325405 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.324388    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs podName:9a90e5c3-5fcb-4016-8682-3803c3121620 nodeName:}" failed. No retries permitted until 2026-09-16 12:00:43.824375016 +0000 UTC m=+3.074912543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs") pod "network-metrics-daemon-bflx7" (UID: "9a90e5c3-5fcb-4016-8682-3803c3121620") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:00:43.325405 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324418    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzrkw\" (UniqueName: \"kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw\") pod \"network-check-target-2ps6l\" (UID: \"becc876e-d896-4811-9ee1-0be40fe69dda\") " pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:00:43.325405 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-run-netns\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.325405 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324516    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fc4ace4-ea76-4d43-8722-093f76f186dd-host-run-netns\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.325405 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324615    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3fc4ace4-ea76-4d43-8722-093f76f186dd-multus-daemon-config\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.325405 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.324870    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c51bc19e-789f-4167-80b8-ada031cdfc55-konnectivity-ca\") pod \"konnectivity-agent-z7wfg\" (UID: \"c51bc19e-789f-4167-80b8-ada031cdfc55\") " pod="kube-system/konnectivity-agent-z7wfg"
Sep 16 12:00:43.326910 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.326890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c51bc19e-789f-4167-80b8-ada031cdfc55-agent-certs\") pod \"konnectivity-agent-z7wfg\" (UID: \"c51bc19e-789f-4167-80b8-ada031cdfc55\") " pod="kube-system/konnectivity-agent-z7wfg"
Sep 16 12:00:43.331013 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.330984    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:00:43.331013 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.331010    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:00:43.331185 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.331036    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lzrkw for pod openshift-network-diagnostics/network-check-target-2ps6l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:00:43.331185 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.331098    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw podName:becc876e-d896-4811-9ee1-0be40fe69dda nodeName:}" failed. No retries permitted until 2026-09-16 12:00:43.831081668 +0000 UTC m=+3.081619195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lzrkw" (UniqueName: "kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw") pod "network-check-target-2ps6l" (UID: "becc876e-d896-4811-9ee1-0be40fe69dda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:00:43.333876 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.333809    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x5nk\" (UniqueName: \"kubernetes.io/projected/3fc4ace4-ea76-4d43-8722-093f76f186dd-kube-api-access-8x5nk\") pod \"multus-5smzb\" (UID: \"3fc4ace4-ea76-4d43-8722-093f76f186dd\") " pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.333876 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.333828    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l96jl\" (UniqueName: \"kubernetes.io/projected/9a90e5c3-5fcb-4016-8682-3803c3121620-kube-api-access-l96jl\") pod \"network-metrics-daemon-bflx7\" (UID: \"9a90e5c3-5fcb-4016-8682-3803c3121620\") " pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:43.406910 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.406873    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w69gx"
Sep 16 12:00:43.414538 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.414511    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2kpnf"
Sep 16 12:00:43.422654 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.422625    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:00:43.428490 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.428469    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq"
Sep 16 12:00:43.435287 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.435254    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-54c4z"
Sep 16 12:00:43.443045 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.443023    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4hzq6"
Sep 16 12:00:43.449900 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.449874    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7q8hz"
Sep 16 12:00:43.456719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.456699    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5smzb"
Sep 16 12:00:43.463567 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.463549    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z7wfg"
Sep 16 12:00:43.827584 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.827484    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs\") pod \"network-metrics-daemon-bflx7\" (UID: \"9a90e5c3-5fcb-4016-8682-3803c3121620\") " pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:43.827779 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.827582    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret\") pod \"global-pull-secret-syncer-8drdm\" (UID: \"bef09744-222b-4d55-9d87-aca589bd4f13\") " pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:43.827779 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.827648    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:00:43.827779 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.827677    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:00:43.827779 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.827746    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs podName:9a90e5c3-5fcb-4016-8682-3803c3121620 nodeName:}" failed. No retries permitted until 2026-09-16 12:00:44.827703828 +0000 UTC m=+4.078241364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs") pod "network-metrics-daemon-bflx7" (UID: "9a90e5c3-5fcb-4016-8682-3803c3121620") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:00:43.827779 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.827773    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret podName:bef09744-222b-4d55-9d87-aca589bd4f13 nodeName:}" failed. No retries permitted until 2026-09-16 12:00:44.827762853 +0000 UTC m=+4.078300382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret") pod "global-pull-secret-syncer-8drdm" (UID: "bef09744-222b-4d55-9d87-aca589bd4f13") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:00:43.882247 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:43.882212    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc4ace4_ea76_4d43_8722_093f76f186dd.slice/crio-a094ed67e1ba8a71efed6fb621569fa5eaed0c018416ac6d5c89895dcbc6912c WatchSource:0}: Error finding container a094ed67e1ba8a71efed6fb621569fa5eaed0c018416ac6d5c89895dcbc6912c: Status 404 returned error can't find the container with id a094ed67e1ba8a71efed6fb621569fa5eaed0c018416ac6d5c89895dcbc6912c
Sep 16 12:00:43.886278 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:43.886255    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5308201_4e49_461a_be86_4c661c0d2783.slice/crio-e9417075819ee441d2b4cab6f17a0110ca9ed03b3ae0a8f9e9b4ddf4ca81cf8f WatchSource:0}: Error finding container e9417075819ee441d2b4cab6f17a0110ca9ed03b3ae0a8f9e9b4ddf4ca81cf8f: Status 404 returned error can't find the container with id e9417075819ee441d2b4cab6f17a0110ca9ed03b3ae0a8f9e9b4ddf4ca81cf8f
Sep 16 12:00:43.887793 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:43.887767    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020357fa_c14e_4e1d_babf_bb7910790e41.slice/crio-27f4a7425d87256a758632e2396d35eddc08bab5b01beffc5d721076f8a83ca8 WatchSource:0}: Error finding container 27f4a7425d87256a758632e2396d35eddc08bab5b01beffc5d721076f8a83ca8: Status 404 returned error can't find the container with id 27f4a7425d87256a758632e2396d35eddc08bab5b01beffc5d721076f8a83ca8
Sep 16 12:00:43.888945 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:43.888923    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b45601_95e3_4427_af82_2358088e7e8a.slice/crio-848c5f697ed86fcc12bbb27ac41282bbb2f72c6766b2a1b63c80e629b1a8470a WatchSource:0}: Error finding container 848c5f697ed86fcc12bbb27ac41282bbb2f72c6766b2a1b63c80e629b1a8470a: Status 404 returned error can't find the container with id 848c5f697ed86fcc12bbb27ac41282bbb2f72c6766b2a1b63c80e629b1a8470a
Sep 16 12:00:43.889804 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:43.889775    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51bc19e_789f_4167_80b8_ada031cdfc55.slice/crio-e8bd057ed80d402ac32946d7f00d7c768bcc3e2b9352b756013810259e1f9db1 WatchSource:0}: Error finding container e8bd057ed80d402ac32946d7f00d7c768bcc3e2b9352b756013810259e1f9db1: Status 404 returned error can't find the container with id e8bd057ed80d402ac32946d7f00d7c768bcc3e2b9352b756013810259e1f9db1
Sep 16 12:00:43.890812 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:43.890668    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff1c306_c904_4cea_8c0b_e0083708fe0f.slice/crio-5849e3e045b1df523659506bf77b61f893f94c0fdc0d590cfeb328baf021132f WatchSource:0}: Error finding container 5849e3e045b1df523659506bf77b61f893f94c0fdc0d590cfeb328baf021132f: Status 404 returned error can't find the container with id 5849e3e045b1df523659506bf77b61f893f94c0fdc0d590cfeb328baf021132f
Sep 16 12:00:43.891896 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:43.891872    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0064df1_27fe_4e46_bc7a_4ef75b8d9892.slice/crio-80d3af0a1dd32c43c1a3044b34fdf0d6c62bc5c982cf3866cd871a586ba6152c WatchSource:0}: Error finding container 80d3af0a1dd32c43c1a3044b34fdf0d6c62bc5c982cf3866cd871a586ba6152c: Status 404 returned error can't find the container with id 80d3af0a1dd32c43c1a3044b34fdf0d6c62bc5c982cf3866cd871a586ba6152c
Sep 16 12:00:43.893038 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:43.892907    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030e2725_fa96_4218_9026_a4a319a47ab4.slice/crio-066969305ba10b15a70ddb13f05e381213b57e2f94326a8b9d5e5305d91a16ba WatchSource:0}: Error finding container 066969305ba10b15a70ddb13f05e381213b57e2f94326a8b9d5e5305d91a16ba: Status 404 returned error can't find the container with id 066969305ba10b15a70ddb13f05e381213b57e2f94326a8b9d5e5305d91a16ba
Sep 16 12:00:43.893887 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:00:43.893722    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25085fb2_c567_4b20_bd8d_6241d7cce1cf.slice/crio-c3291621bf634bb9325c341e1db16f697b51f888dcd24b161900026df00c65cd WatchSource:0}: Error finding container c3291621bf634bb9325c341e1db16f697b51f888dcd24b161900026df00c65cd: Status 404 returned error can't find the container with id c3291621bf634bb9325c341e1db16f697b51f888dcd24b161900026df00c65cd
Sep 16 12:00:43.928351 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:43.928328    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzrkw\" (UniqueName: \"kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw\") pod \"network-check-target-2ps6l\" (UID: \"becc876e-d896-4811-9ee1-0be40fe69dda\") " pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:00:43.928486 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.928471    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:00:43.928535 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.928496    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:00:43.928535 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.928508    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lzrkw for pod openshift-network-diagnostics/network-check-target-2ps6l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:00:43.928630 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:43.928563    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw podName:becc876e-d896-4811-9ee1-0be40fe69dda nodeName:}" failed. No retries permitted until 2026-09-16 12:00:44.928545736 +0000 UTC m=+4.179083264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lzrkw" (UniqueName: "kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw") pod "network-check-target-2ps6l" (UID: "becc876e-d896-4811-9ee1-0be40fe69dda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:00:44.149575 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:44.149492    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 11:55:43 +0000 UTC" deadline="2028-02-17 18:51:34.171861717 +0000 UTC"
Sep 16 12:00:44.149575 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:44.149521    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12462h50m50.022345837s"
Sep 16 12:00:44.206013 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:44.205986    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:44.206462 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:44.206129    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:00:44.216269 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:44.216231    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal" event={"ID":"daddab1c0f05fb32f4f8c0ebec22f11c","Type":"ContainerStarted","Data":"b4c7542b75be4bfb907f4b60804c9c34bb963eb7abd4389954d2fa7ac682f8cc"}
Sep 16 12:00:44.222017 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:44.221986    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-54c4z" event={"ID":"c0064df1-27fe-4e46-bc7a-4ef75b8d9892","Type":"ContainerStarted","Data":"80d3af0a1dd32c43c1a3044b34fdf0d6c62bc5c982cf3866cd871a586ba6152c"}
Sep 16 12:00:44.223399 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:44.223371    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq" event={"ID":"5ff1c306-c904-4cea-8c0b-e0083708fe0f","Type":"ContainerStarted","Data":"5849e3e045b1df523659506bf77b61f893f94c0fdc0d590cfeb328baf021132f"}
Sep 16 12:00:44.225459 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:44.225433    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2kpnf" event={"ID":"020357fa-c14e-4e1d-babf-bb7910790e41","Type":"ContainerStarted","Data":"27f4a7425d87256a758632e2396d35eddc08bab5b01beffc5d721076f8a83ca8"}
Sep 16 12:00:44.226692 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:44.226668    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5smzb" event={"ID":"3fc4ace4-ea76-4d43-8722-093f76f186dd","Type":"ContainerStarted","Data":"a094ed67e1ba8a71efed6fb621569fa5eaed0c018416ac6d5c89895dcbc6912c"}
Sep 16 12:00:44.227984 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:44.227955    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w69gx" event={"ID":"25085fb2-c567-4b20-bd8d-6241d7cce1cf","Type":"ContainerStarted","Data":"c3291621bf634bb9325c341e1db16f697b51f888dcd24b161900026df00c65cd"}
Sep 16 12:00:44.229265 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:44.229239    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d" event={"ID":"030e2725-fa96-4218-9026-a4a319a47ab4","Type":"ContainerStarted","Data":"066969305ba10b15a70ddb13f05e381213b57e2f94326a8b9d5e5305d91a16ba"}
Sep 16 12:00:44.230720 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:44.230554    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z7wfg" event={"ID":"c51bc19e-789f-4167-80b8-ada031cdfc55","Type":"ContainerStarted","Data":"e8bd057ed80d402ac32946d7f00d7c768bcc3e2b9352b756013810259e1f9db1"}
Sep 16 12:00:44.231189 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:44.231134    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal" podStartSLOduration=2.231117882 podStartE2EDuration="2.231117882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:00:44.230959006 +0000 UTC m=+3.481496555" watchObservedRunningTime="2026-09-16 12:00:44.231117882 +0000 UTC m=+3.481655431"
Sep 16 12:00:44.231847 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:44.231817    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7q8hz" event={"ID":"55b45601-95e3-4427-af82-2358088e7e8a","Type":"ContainerStarted","Data":"848c5f697ed86fcc12bbb27ac41282bbb2f72c6766b2a1b63c80e629b1a8470a"}
Sep 16 12:00:44.232909 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:44.232884    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4hzq6" event={"ID":"e5308201-4e49-461a-be86-4c661c0d2783","Type":"ContainerStarted","Data":"e9417075819ee441d2b4cab6f17a0110ca9ed03b3ae0a8f9e9b4ddf4ca81cf8f"}
Sep 16 12:00:44.837321 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:44.837041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret\") pod \"global-pull-secret-syncer-8drdm\" (UID: \"bef09744-222b-4d55-9d87-aca589bd4f13\") " pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:44.837321 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:44.837115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs\") pod \"network-metrics-daemon-bflx7\" (UID: \"9a90e5c3-5fcb-4016-8682-3803c3121620\") " pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:44.837321 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:44.837240    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:00:44.837321 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:44.837307    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs podName:9a90e5c3-5fcb-4016-8682-3803c3121620 nodeName:}" failed. No retries permitted until 2026-09-16 12:00:46.837286509 +0000 UTC m=+6.087824053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs") pod "network-metrics-daemon-bflx7" (UID: "9a90e5c3-5fcb-4016-8682-3803c3121620") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:00:44.837544 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:44.837381    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:00:44.837544 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:44.837415    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret podName:bef09744-222b-4d55-9d87-aca589bd4f13 nodeName:}" failed. No retries permitted until 2026-09-16 12:00:46.837404379 +0000 UTC m=+6.087941919 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret") pod "global-pull-secret-syncer-8drdm" (UID: "bef09744-222b-4d55-9d87-aca589bd4f13") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:00:44.938321 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:44.938276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzrkw\" (UniqueName: \"kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw\") pod \"network-check-target-2ps6l\" (UID: \"becc876e-d896-4811-9ee1-0be40fe69dda\") " pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:00:44.938513 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:44.938453    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:00:44.938513 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:44.938477    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:00:44.938513 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:44.938490    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lzrkw for pod openshift-network-diagnostics/network-check-target-2ps6l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:00:44.938686 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:44.938548    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw podName:becc876e-d896-4811-9ee1-0be40fe69dda nodeName:}" failed. No retries permitted until 2026-09-16 12:00:46.938531203 +0000 UTC m=+6.189068733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lzrkw" (UniqueName: "kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw") pod "network-check-target-2ps6l" (UID: "becc876e-d896-4811-9ee1-0be40fe69dda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:00:45.150050 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:45.149898    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 11:55:43 +0000 UTC" deadline="2028-02-26 03:01:02.767868492 +0000 UTC"
Sep 16 12:00:45.150050 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:45.149964    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12663h0m17.617908438s"
Sep 16 12:00:45.208215 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:45.208181    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:45.208374 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:45.208334    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:00:45.208894 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:45.208751    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:00:45.208894 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:45.208849    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ps6l" podUID="becc876e-d896-4811-9ee1-0be40fe69dda"
Sep 16 12:00:46.206361 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:46.206323    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:46.206870 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:46.206466    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:00:46.853272 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:46.853234    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret\") pod \"global-pull-secret-syncer-8drdm\" (UID: \"bef09744-222b-4d55-9d87-aca589bd4f13\") " pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:46.853484 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:46.853304    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs\") pod \"network-metrics-daemon-bflx7\" (UID: \"9a90e5c3-5fcb-4016-8682-3803c3121620\") " pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:46.853484 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:46.853456    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:00:46.853605 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:46.853517    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs podName:9a90e5c3-5fcb-4016-8682-3803c3121620 nodeName:}" failed. No retries permitted until 2026-09-16 12:00:50.853499537 +0000 UTC m=+10.104037068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs") pod "network-metrics-daemon-bflx7" (UID: "9a90e5c3-5fcb-4016-8682-3803c3121620") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:00:46.853931 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:46.853810    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:00:46.853931 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:46.853898    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret podName:bef09744-222b-4d55-9d87-aca589bd4f13 nodeName:}" failed. No retries permitted until 2026-09-16 12:00:50.853878083 +0000 UTC m=+10.104415631 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret") pod "global-pull-secret-syncer-8drdm" (UID: "bef09744-222b-4d55-9d87-aca589bd4f13") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:00:46.954704 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:46.954097    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzrkw\" (UniqueName: \"kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw\") pod \"network-check-target-2ps6l\" (UID: \"becc876e-d896-4811-9ee1-0be40fe69dda\") " pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:00:46.954704 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:46.954274    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:00:46.954704 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:46.954293    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:00:46.954704 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:46.954306    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lzrkw for pod openshift-network-diagnostics/network-check-target-2ps6l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:00:46.954704 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:46.954364    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw podName:becc876e-d896-4811-9ee1-0be40fe69dda nodeName:}" failed. No retries permitted until 2026-09-16 12:00:50.954346382 +0000 UTC m=+10.204883911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lzrkw" (UniqueName: "kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw") pod "network-check-target-2ps6l" (UID: "becc876e-d896-4811-9ee1-0be40fe69dda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:00:47.208627 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:47.208552    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:00:47.209097 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:47.208682    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ps6l" podUID="becc876e-d896-4811-9ee1-0be40fe69dda"
Sep 16 12:00:47.209097 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:47.209085    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:47.209214 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:47.209170    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:00:48.206805 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:48.206295    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:48.206805 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:48.206432    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:00:49.206530 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:49.206497    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:49.207034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:49.206500    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:00:49.207034 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:49.206655    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:00:49.207034 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:49.206704    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ps6l" podUID="becc876e-d896-4811-9ee1-0be40fe69dda"
Sep 16 12:00:50.207058 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:50.206718    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:50.207465 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:50.207131    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:00:50.891064 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:50.891017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret\") pod \"global-pull-secret-syncer-8drdm\" (UID: \"bef09744-222b-4d55-9d87-aca589bd4f13\") " pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:50.891250 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:50.891094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs\") pod \"network-metrics-daemon-bflx7\" (UID: \"9a90e5c3-5fcb-4016-8682-3803c3121620\") " pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:50.891250 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:50.891192    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:00:50.891250 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:50.891224    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:00:50.891429 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:50.891272    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret podName:bef09744-222b-4d55-9d87-aca589bd4f13 nodeName:}" failed. No retries permitted until 2026-09-16 12:00:58.891249951 +0000 UTC m=+18.141787500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret") pod "global-pull-secret-syncer-8drdm" (UID: "bef09744-222b-4d55-9d87-aca589bd4f13") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:00:50.891429 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:50.891297    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs podName:9a90e5c3-5fcb-4016-8682-3803c3121620 nodeName:}" failed. No retries permitted until 2026-09-16 12:00:58.891285076 +0000 UTC m=+18.141822606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs") pod "network-metrics-daemon-bflx7" (UID: "9a90e5c3-5fcb-4016-8682-3803c3121620") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:00:50.991993 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:50.991957    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzrkw\" (UniqueName: \"kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw\") pod \"network-check-target-2ps6l\" (UID: \"becc876e-d896-4811-9ee1-0be40fe69dda\") " pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:00:50.992172 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:50.992146    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:00:50.992172 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:50.992174    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:00:50.992365 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:50.992185    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lzrkw for pod openshift-network-diagnostics/network-check-target-2ps6l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:00:50.992365 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:50.992247    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw podName:becc876e-d896-4811-9ee1-0be40fe69dda nodeName:}" failed. No retries permitted until 2026-09-16 12:00:58.992231595 +0000 UTC m=+18.242769126 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lzrkw" (UniqueName: "kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw") pod "network-check-target-2ps6l" (UID: "becc876e-d896-4811-9ee1-0be40fe69dda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:00:51.208036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:51.208002    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:51.208450 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:51.208131    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:00:51.208519 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:51.208482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:00:51.208578 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:51.208552    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ps6l" podUID="becc876e-d896-4811-9ee1-0be40fe69dda"
Sep 16 12:00:52.206600 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:52.206565    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:52.206796 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:52.206697    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:00:53.206402 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:53.205918    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:53.206402 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:53.206063    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:00:53.206402 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:53.206111    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:00:53.206402 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:53.206232    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ps6l" podUID="becc876e-d896-4811-9ee1-0be40fe69dda"
Sep 16 12:00:54.206233 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:54.206148    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:54.206416 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:54.206277    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:00:55.206696 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:55.206605    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:00:55.207114 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:55.206722    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ps6l" podUID="becc876e-d896-4811-9ee1-0be40fe69dda"
Sep 16 12:00:55.207114 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:55.206606    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:55.207267 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:55.207147    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:00:56.205959 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:56.205924    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:56.206159 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:56.206070    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:00:57.210079 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:57.210044    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:00:57.210617 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:57.210205    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ps6l" podUID="becc876e-d896-4811-9ee1-0be40fe69dda"
Sep 16 12:00:57.210617 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:57.210272    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:57.210617 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:57.210407    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:00:58.206368 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:58.205854    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:58.206368 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:58.205987    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:00:58.954286 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:58.954192    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs\") pod \"network-metrics-daemon-bflx7\" (UID: \"9a90e5c3-5fcb-4016-8682-3803c3121620\") " pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:58.954286 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:58.954287    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret\") pod \"global-pull-secret-syncer-8drdm\" (UID: \"bef09744-222b-4d55-9d87-aca589bd4f13\") " pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:00:58.954759 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:58.954359    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:00:58.954759 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:58.954411    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:00:58.954759 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:58.954428    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs podName:9a90e5c3-5fcb-4016-8682-3803c3121620 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:14.954411614 +0000 UTC m=+34.204949167 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs") pod "network-metrics-daemon-bflx7" (UID: "9a90e5c3-5fcb-4016-8682-3803c3121620") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:00:58.954759 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:58.954463    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret podName:bef09744-222b-4d55-9d87-aca589bd4f13 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:14.95444612 +0000 UTC m=+34.204983661 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret") pod "global-pull-secret-syncer-8drdm" (UID: "bef09744-222b-4d55-9d87-aca589bd4f13") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:00:59.055430 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:59.055390    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzrkw\" (UniqueName: \"kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw\") pod \"network-check-target-2ps6l\" (UID: \"becc876e-d896-4811-9ee1-0be40fe69dda\") " pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:00:59.055605 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:59.055590    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:00:59.055691 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:59.055616    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:00:59.055691 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:59.055630    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lzrkw for pod openshift-network-diagnostics/network-check-target-2ps6l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:00:59.055807 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:59.055697    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw podName:becc876e-d896-4811-9ee1-0be40fe69dda nodeName:}" failed. No retries permitted until 2026-09-16 12:01:15.055675562 +0000 UTC m=+34.306213110 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lzrkw" (UniqueName: "kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw") pod "network-check-target-2ps6l" (UID: "becc876e-d896-4811-9ee1-0be40fe69dda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:00:59.206081 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:59.205996    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:00:59.206081 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:00:59.206027    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:00:59.206300 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:59.206127    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:00:59.206599 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:00:59.206568    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ps6l" podUID="becc876e-d896-4811-9ee1-0be40fe69dda"
Sep 16 12:01:00.206561 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:00.206522    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:01:00.206988 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:00.206657    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:01:01.206991 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:01.206951    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:01:01.207438 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:01.207079    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ps6l" podUID="becc876e-d896-4811-9ee1-0be40fe69dda"
Sep 16 12:01:01.207438 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:01.207139    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:01:01.207438 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:01.207267    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:01:02.206355 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:02.205847    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:01:02.206355 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:02.205974    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:01:02.284888 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:02.284207    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5smzb" event={"ID":"3fc4ace4-ea76-4d43-8722-093f76f186dd","Type":"ContainerStarted","Data":"762a4ad348e62ebe9716cb32c5ba8831ac562ee7d4f3f5d523d42292a469965b"}
Sep 16 12:01:02.297202 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:02.297112    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d" event={"ID":"030e2725-fa96-4218-9026-a4a319a47ab4","Type":"ContainerStarted","Data":"17a38c43f3f1fe081cc5716d0011468bc37b92803460a89d425ed7ecf7f806ae"}
Sep 16 12:01:02.308630 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:02.308505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-54c4z" event={"ID":"c0064df1-27fe-4e46-bc7a-4ef75b8d9892","Type":"ContainerStarted","Data":"a658170ba5a81f185ad1ab8baf27e8e3443d64564ec0df996f5ee943fdc3c15a"}
Sep 16 12:01:02.334128 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:02.332883    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5smzb" podStartSLOduration=3.073486741 podStartE2EDuration="21.332861413s" totalImagesPullingTime="18.259374672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:01:02.303609112 +0000 UTC m=+21.554146662" watchObservedRunningTime="2026-09-16 12:01:02.332861413 +0000 UTC m=+21.583398962"
Sep 16 12:01:03.206748 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.206709    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:01:03.206921 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:03.206846    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:01:03.206921 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.206906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:01:03.207011 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:03.206997    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ps6l" podUID="becc876e-d896-4811-9ee1-0be40fe69dda"
Sep 16 12:01:03.312946 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.312910    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq" event={"ID":"5ff1c306-c904-4cea-8c0b-e0083708fe0f","Type":"ContainerStarted","Data":"80e27751b6e5afa539378ee70a98a704ebc78fd12d50d8370625c5d5e9df9bee"}
Sep 16 12:01:03.315080 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.315053    2582 generic.go:356] "Generic (PLEG): container finished" podID="623b25dd964a2b389941b772b9ff3ebf" containerID="cc91be11de9505f641dcece36af3f6e6f636dbff0f61ac166ed2a10366e59814" exitCode=0
Sep 16 12:01:03.315207 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.315138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal" event={"ID":"623b25dd964a2b389941b772b9ff3ebf","Type":"ContainerDied","Data":"cc91be11de9505f641dcece36af3f6e6f636dbff0f61ac166ed2a10366e59814"}
Sep 16 12:01:03.317128 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.317099    2582 generic.go:356] "Generic (PLEG): container finished" podID="25085fb2-c567-4b20-bd8d-6241d7cce1cf" containerID="b960cfe3c31be8cde2989f370fe28b1498752f5f377a15005899b375700c433f" exitCode=0
Sep 16 12:01:03.317238 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.317145    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w69gx" event={"ID":"25085fb2-c567-4b20-bd8d-6241d7cce1cf","Type":"ContainerDied","Data":"b960cfe3c31be8cde2989f370fe28b1498752f5f377a15005899b375700c433f"}
Sep 16 12:01:03.321577 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.321550    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d" event={"ID":"030e2725-fa96-4218-9026-a4a319a47ab4","Type":"ContainerStarted","Data":"005b4785c12b4a55c45fb1fc899e77dc436fe3faba632b487fada4eabfac239a"}
Sep 16 12:01:03.321690 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.321642    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d" event={"ID":"030e2725-fa96-4218-9026-a4a319a47ab4","Type":"ContainerStarted","Data":"e769c4258b61816ebd3871a151172fcd614bfa9cef5d12455badd53da69be324"}
Sep 16 12:01:03.321775 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.321729    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d" event={"ID":"030e2725-fa96-4218-9026-a4a319a47ab4","Type":"ContainerStarted","Data":"67f141479e476900b889bbadac2a3d3e3a7ec652c34a7eb5876ddf0eb927ee59"}
Sep 16 12:01:03.321840 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.321797    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d" event={"ID":"030e2725-fa96-4218-9026-a4a319a47ab4","Type":"ContainerStarted","Data":"486b3693ffaf61af5b3c239e666140d237bcd42512261955313e085381391020"}
Sep 16 12:01:03.321907 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.321849    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d" event={"ID":"030e2725-fa96-4218-9026-a4a319a47ab4","Type":"ContainerStarted","Data":"30a3598c47c9b9b9e9375c4689072e486dcc829c134108f73b5023d852821ba7"}
Sep 16 12:01:03.323387 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.323359    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z7wfg" event={"ID":"c51bc19e-789f-4167-80b8-ada031cdfc55","Type":"ContainerStarted","Data":"46825be2e0a933109ab2775858beea3e1e9cb87c2566e9102d4bff65264cdf3a"}
Sep 16 12:01:03.325621 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.325600    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7q8hz" event={"ID":"55b45601-95e3-4427-af82-2358088e7e8a","Type":"ContainerStarted","Data":"b3e68a980db0312637618a6a0a50860e9903677e8cc7f4921206a202c6e131f9"}
Sep 16 12:01:03.327421 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.327397    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4hzq6" event={"ID":"e5308201-4e49-461a-be86-4c661c0d2783","Type":"ContainerStarted","Data":"64410d721116821a9dcf44ae5fc1b096cf03788d7e26eaebc1fc446fdd3c701a"}
Sep 16 12:01:03.334517 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.334473    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-54c4z" podStartSLOduration=4.15030198 podStartE2EDuration="22.3344585s" totalImagesPullingTime="18.18415652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:01:02.334887373 +0000 UTC m=+21.585424922" watchObservedRunningTime="2026-09-16 12:01:03.3344585 +0000 UTC m=+22.584996104"
Sep 16 12:01:03.353170 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.352933    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z7wfg" podStartSLOduration=7.709434614 podStartE2EDuration="22.352914943s" totalImagesPullingTime="14.643480329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:01:03.352583058 +0000 UTC m=+22.603120621" watchObservedRunningTime="2026-09-16 12:01:03.352914943 +0000 UTC m=+22.603452511"
Sep 16 12:01:03.416871 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.416819    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7q8hz" podStartSLOduration=4.218563354 podStartE2EDuration="22.416794386s" totalImagesPullingTime="18.198231032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:01:03.392504598 +0000 UTC m=+22.643042148" watchObservedRunningTime="2026-09-16 12:01:03.416794386 +0000 UTC m=+22.667331936"
Sep 16 12:01:03.417656 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.417612    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4hzq6" podStartSLOduration=4.223708171 podStartE2EDuration="22.417593986s" totalImagesPullingTime="18.193885815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:01:03.416871712 +0000 UTC m=+22.667409261" watchObservedRunningTime="2026-09-16 12:01:03.417593986 +0000 UTC m=+22.668131535"
Sep 16 12:01:03.442228 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.442109    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z7wfg"
Sep 16 12:01:03.442922 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.442902    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z7wfg"
Sep 16 12:01:03.892579 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:03.892553    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 12:01:04.184493 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:04.184344    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T12:01:03.892573571Z","UUID":"c36c947c-5de8-4293-b61d-4388564c96e7","Handler":null,"Name":"","Endpoint":""}
Sep 16 12:01:04.187428 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:04.187401    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 12:01:04.187584 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:04.187443    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 12:01:04.205665 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:04.205633    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:01:04.205873 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:04.205724    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:01:04.331292 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:04.331257    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal" event={"ID":"623b25dd964a2b389941b772b9ff3ebf","Type":"ContainerStarted","Data":"91fe204ac606f96d55d36bbfa24901e8633c4871e1ed5b39882e5569cc9c8fa9"}
Sep 16 12:01:04.333117 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:04.333089    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq" event={"ID":"5ff1c306-c904-4cea-8c0b-e0083708fe0f","Type":"ContainerStarted","Data":"dbfc6601122533e83245b0273dfc9727f34e7c91a2dd6a295b0bdeac18daa0fd"}
Sep 16 12:01:04.334525 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:04.334495    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2kpnf" event={"ID":"020357fa-c14e-4e1d-babf-bb7910790e41","Type":"ContainerStarted","Data":"2d67bb43ea1f42c4a97781124b39137540d6abf92e165f831962bfc717fc757b"}
Sep 16 12:01:04.346327 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:04.346276    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal" podStartSLOduration=22.346259089 podStartE2EDuration="22.346259089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:01:04.345698753 +0000 UTC m=+23.596236303" watchObservedRunningTime="2026-09-16 12:01:04.346259089 +0000 UTC m=+23.596796641"
Sep 16 12:01:04.359430 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:04.359389    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2kpnf" podStartSLOduration=5.156439057 podStartE2EDuration="23.359377547s" totalImagesPullingTime="18.20293849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:01:04.359092247 +0000 UTC m=+23.609629793" watchObservedRunningTime="2026-09-16 12:01:04.359377547 +0000 UTC m=+23.609915095"
Sep 16 12:01:05.205853 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:05.205767    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:01:05.205853 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:05.205809    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:01:05.206071 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:05.205919    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:01:05.206308 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:05.206277    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ps6l" podUID="becc876e-d896-4811-9ee1-0be40fe69dda"
Sep 16 12:01:05.338892 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:05.338859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq" event={"ID":"5ff1c306-c904-4cea-8c0b-e0083708fe0f","Type":"ContainerStarted","Data":"d0b66afee249a9a21413de5d830b2d75370ed7fa61280937ddf66089100ad754"}
Sep 16 12:01:05.342368 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:05.342340    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 12:01:05.342368 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:05.342344    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d" event={"ID":"030e2725-fa96-4218-9026-a4a319a47ab4","Type":"ContainerStarted","Data":"077dd6870e8a7c69b35dbd5e0d231a0c804fe6ae762510965c9c4d543631d4b3"}
Sep 16 12:01:05.356376 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:05.356327    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqpxq" podStartSLOduration=3.870523458 podStartE2EDuration="24.356307478s" totalImagesPullingTime="20.48578402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:00:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:01:05.355792221 +0000 UTC m=+24.606329773" watchObservedRunningTime="2026-09-16 12:01:05.356307478 +0000 UTC m=+24.606845031"
Sep 16 12:01:06.206296 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:06.206245    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:01:06.206477 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:06.206394    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:01:07.206582 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:07.206536    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:01:07.206582 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:07.206575    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:01:07.207047 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:07.206699    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:01:07.207047 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:07.206834    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ps6l" podUID="becc876e-d896-4811-9ee1-0be40fe69dda"
Sep 16 12:01:08.206235 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:08.206189    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:01:08.206395 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:08.206308    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:01:09.206592 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:09.206350    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:01:09.207229 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:09.206350    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:01:09.207229 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:09.206640    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:01:09.207229 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:09.206680    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ps6l" podUID="becc876e-d896-4811-9ee1-0be40fe69dda"
Sep 16 12:01:09.359604 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:09.359561    2582 generic.go:356] "Generic (PLEG): container finished" podID="25085fb2-c567-4b20-bd8d-6241d7cce1cf" containerID="65852bcc1d72527b093e82ea4829e1e8628bb0a56d9121175964d8250cab0084" exitCode=0
Sep 16 12:01:09.359792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:09.359642    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w69gx" event={"ID":"25085fb2-c567-4b20-bd8d-6241d7cce1cf","Type":"ContainerDied","Data":"65852bcc1d72527b093e82ea4829e1e8628bb0a56d9121175964d8250cab0084"}
Sep 16 12:01:09.363407 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:09.363383    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d" event={"ID":"030e2725-fa96-4218-9026-a4a319a47ab4","Type":"ContainerStarted","Data":"5c836e5d90d9e121d8091e83dbb0d3a0cd58d17da2f8eb879a1e6df0cb60c37a"}
Sep 16 12:01:09.363594 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:09.363578    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:01:09.363651 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:09.363633    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:01:09.379552 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:09.379531    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:01:09.410237 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:09.410174    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d" podStartSLOduration=10.178956513 podStartE2EDuration="28.410157582s" totalImagesPullingTime="18.231201069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:01:09.409592361 +0000 UTC m=+28.660129909" watchObservedRunningTime="2026-09-16 12:01:09.410157582 +0000 UTC m=+28.660695131"
Sep 16 12:01:10.206305 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:10.206052    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:01:10.206503 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:10.206419    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:01:10.238398 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:10.238362    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bflx7"]
Sep 16 12:01:10.239008 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:10.238518    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:01:10.239008 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:10.238644    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:01:10.241225 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:10.241196    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8drdm"]
Sep 16 12:01:10.241756 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:10.241720    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2ps6l"]
Sep 16 12:01:10.241864 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:10.241839    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:01:10.241956 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:10.241937    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ps6l" podUID="becc876e-d896-4811-9ee1-0be40fe69dda"
Sep 16 12:01:10.365723 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:10.365612    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:01:10.365880 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:10.365774    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:01:10.366097 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:10.366081    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:01:10.383222 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:10.383183    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:01:11.027792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:11.027763    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z7wfg"
Sep 16 12:01:11.027935 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:11.027875    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 12:01:11.028411 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:11.028387    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z7wfg"
Sep 16 12:01:11.370413 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:11.370381    2582 generic.go:356] "Generic (PLEG): container finished" podID="25085fb2-c567-4b20-bd8d-6241d7cce1cf" containerID="205685ea2a65687d51f958c13c539b7a54b4e7e5b71fa0de666a8b58eaa1a5b8" exitCode=0
Sep 16 12:01:11.370873 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:11.370479    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w69gx" event={"ID":"25085fb2-c567-4b20-bd8d-6241d7cce1cf","Type":"ContainerDied","Data":"205685ea2a65687d51f958c13c539b7a54b4e7e5b71fa0de666a8b58eaa1a5b8"}
Sep 16 12:01:12.205900 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:12.205861    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:01:12.205900 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:12.205898    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:01:12.206066 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:12.205861    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:01:12.206066 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:12.205993    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ps6l" podUID="becc876e-d896-4811-9ee1-0be40fe69dda"
Sep 16 12:01:12.206066 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:12.206045    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:01:12.206165 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:12.206134    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:01:13.378123 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:13.378093    2582 generic.go:356] "Generic (PLEG): container finished" podID="25085fb2-c567-4b20-bd8d-6241d7cce1cf" containerID="469f83b2085f3230625b77f11a49a8f89a544f65e19d64536f9fed0fface5cd3" exitCode=0
Sep 16 12:01:13.378123 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:13.378128    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w69gx" event={"ID":"25085fb2-c567-4b20-bd8d-6241d7cce1cf","Type":"ContainerDied","Data":"469f83b2085f3230625b77f11a49a8f89a544f65e19d64536f9fed0fface5cd3"}
Sep 16 12:01:14.206171 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:14.206135    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:01:14.206171 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:14.206168    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:01:14.206486 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:14.206168    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:01:14.206486 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:14.206270    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:01:14.206486 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:14.206327    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8drdm" podUID="bef09744-222b-4d55-9d87-aca589bd4f13"
Sep 16 12:01:14.206486 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:14.206401    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ps6l" podUID="becc876e-d896-4811-9ee1-0be40fe69dda"
Sep 16 12:01:14.971850 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:14.971761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret\") pod \"global-pull-secret-syncer-8drdm\" (UID: \"bef09744-222b-4d55-9d87-aca589bd4f13\") " pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:01:14.971850 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:14.971811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs\") pod \"network-metrics-daemon-bflx7\" (UID: \"9a90e5c3-5fcb-4016-8682-3803c3121620\") " pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:01:14.972510 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:14.971933    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:14.972510 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:14.971947    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:14.972510 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:14.971997    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs podName:9a90e5c3-5fcb-4016-8682-3803c3121620 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:46.971976782 +0000 UTC m=+66.222514330 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs") pod "network-metrics-daemon-bflx7" (UID: "9a90e5c3-5fcb-4016-8682-3803c3121620") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:14.972510 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:14.972018    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret podName:bef09744-222b-4d55-9d87-aca589bd4f13 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:46.972008495 +0000 UTC m=+66.222546027 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret") pod "global-pull-secret-syncer-8drdm" (UID: "bef09744-222b-4d55-9d87-aca589bd4f13") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:15.072960 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.072917    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzrkw\" (UniqueName: \"kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw\") pod \"network-check-target-2ps6l\" (UID: \"becc876e-d896-4811-9ee1-0be40fe69dda\") " pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:01:15.073234 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:15.073086    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:01:15.073234 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:15.073109    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:01:15.073234 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:15.073120    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lzrkw for pod openshift-network-diagnostics/network-check-target-2ps6l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:15.073234 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:15.073178    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw podName:becc876e-d896-4811-9ee1-0be40fe69dda nodeName:}" failed. No retries permitted until 2026-09-16 12:01:47.073159183 +0000 UTC m=+66.323696731 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lzrkw" (UniqueName: "kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw") pod "network-check-target-2ps6l" (UID: "becc876e-d896-4811-9ee1-0be40fe69dda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:15.074461 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.074440    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeReady"
Sep 16 12:01:15.074609 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.074598    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 12:01:15.106099 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.106068    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59d44784f4-6mqnw"]
Sep 16 12:01:15.120717 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.120681    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-t8wnw"]
Sep 16 12:01:15.120901 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.120835    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.123408 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.123383    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 12:01:15.123869 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.123604    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 12:01:15.124495 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.124477    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pr5zs\""
Sep 16 12:01:15.124669 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.124650    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 12:01:15.129903 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.129538    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 12:01:15.134813 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.134794    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wf5rh"]
Sep 16 12:01:15.134918 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.134903    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t8wnw"
Sep 16 12:01:15.137799 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.137781    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 12:01:15.137913 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.137823    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 12:01:15.138079 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.137994    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 12:01:15.138180 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.138118    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-knmsl\""
Sep 16 12:01:15.154834 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.154813    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59d44784f4-6mqnw"]
Sep 16 12:01:15.154939 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.154847    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wf5rh"]
Sep 16 12:01:15.154939 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.154874    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t8wnw"]
Sep 16 12:01:15.155005 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.154938    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:15.157382 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.157354    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 12:01:15.157597 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.157578    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rfzsk\""
Sep 16 12:01:15.157723 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.157636    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 12:01:15.274107 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.274076    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84e4fef2-7c44-4c75-a75a-20fc52336320-cert\") pod \"ingress-canary-t8wnw\" (UID: \"84e4fef2-7c44-4c75-a75a-20fc52336320\") " pod="openshift-ingress-canary/ingress-canary-t8wnw"
Sep 16 12:01:15.274229 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.274113    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttmq5\" (UniqueName: \"kubernetes.io/projected/84e4fef2-7c44-4c75-a75a-20fc52336320-kube-api-access-ttmq5\") pod \"ingress-canary-t8wnw\" (UID: \"84e4fef2-7c44-4c75-a75a-20fc52336320\") " pod="openshift-ingress-canary/ingress-canary-t8wnw"
Sep 16 12:01:15.274229 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.274138    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-config-volume\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:15.274229 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.274156    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/751c95c1-cd17-48f4-9d07-8c7c6c312435-image-registry-private-configuration\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.274229 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.274176    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/751c95c1-cd17-48f4-9d07-8c7c6c312435-ca-trust-extracted\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.274229 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.274199    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-bound-sa-token\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.274229 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.274223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jcwg\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-kube-api-access-8jcwg\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.274459 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.274247    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.274459 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.274277    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:15.274459 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.274299    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-tmp-dir\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:15.274459 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.274315    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-certificates\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.274459 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.274334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/751c95c1-cd17-48f4-9d07-8c7c6c312435-trusted-ca\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.274459 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.274364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ks5b\" (UniqueName: \"kubernetes.io/projected/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-kube-api-access-8ks5b\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:15.274459 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.274408    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/751c95c1-cd17-48f4-9d07-8c7c6c312435-installation-pull-secrets\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.375433 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.375396    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84e4fef2-7c44-4c75-a75a-20fc52336320-cert\") pod \"ingress-canary-t8wnw\" (UID: \"84e4fef2-7c44-4c75-a75a-20fc52336320\") " pod="openshift-ingress-canary/ingress-canary-t8wnw"
Sep 16 12:01:15.375433 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.375430    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttmq5\" (UniqueName: \"kubernetes.io/projected/84e4fef2-7c44-4c75-a75a-20fc52336320-kube-api-access-ttmq5\") pod \"ingress-canary-t8wnw\" (UID: \"84e4fef2-7c44-4c75-a75a-20fc52336320\") " pod="openshift-ingress-canary/ingress-canary-t8wnw"
Sep 16 12:01:15.375693 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.375463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-config-volume\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:15.375693 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.375491    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/751c95c1-cd17-48f4-9d07-8c7c6c312435-image-registry-private-configuration\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.375693 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.375520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/751c95c1-cd17-48f4-9d07-8c7c6c312435-ca-trust-extracted\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.375693 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.375541    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-bound-sa-token\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.375693 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.375563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jcwg\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-kube-api-access-8jcwg\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.375693 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.375592    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.375693 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.375627    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:15.375693 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.375661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-tmp-dir\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:15.375693 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.375683    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-certificates\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.376179 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.375723    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/751c95c1-cd17-48f4-9d07-8c7c6c312435-trusted-ca\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.376179 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.375772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ks5b\" (UniqueName: \"kubernetes.io/projected/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-kube-api-access-8ks5b\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:15.376179 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.375809    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/751c95c1-cd17-48f4-9d07-8c7c6c312435-installation-pull-secrets\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.376179 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.376154    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-config-volume\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:15.376179 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.376169    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-tmp-dir\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:15.376506 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:15.376483    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:01:15.376506 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:15.376503    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d44784f4-6mqnw: secret "image-registry-tls" not found
Sep 16 12:01:15.376593 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:15.376557    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls podName:751c95c1-cd17-48f4-9d07-8c7c6c312435 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:15.876540205 +0000 UTC m=+35.127077735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls") pod "image-registry-59d44784f4-6mqnw" (UID: "751c95c1-cd17-48f4-9d07-8c7c6c312435") : secret "image-registry-tls" not found
Sep 16 12:01:15.376904 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:15.376845    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:01:15.376904 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:15.376893    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls podName:aab5b1d1-1603-47a0-a5cc-0d136c7f678a nodeName:}" failed. No retries permitted until 2026-09-16 12:01:15.876878251 +0000 UTC m=+35.127415785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls") pod "dns-default-wf5rh" (UID: "aab5b1d1-1603-47a0-a5cc-0d136c7f678a") : secret "dns-default-metrics-tls" not found
Sep 16 12:01:15.377087 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.377061    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/751c95c1-cd17-48f4-9d07-8c7c6c312435-trusted-ca\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.377405 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.377382    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-certificates\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.377551 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.377531    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/751c95c1-cd17-48f4-9d07-8c7c6c312435-ca-trust-extracted\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.380590 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.380480    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/751c95c1-cd17-48f4-9d07-8c7c6c312435-installation-pull-secrets\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.380667 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.380487    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/751c95c1-cd17-48f4-9d07-8c7c6c312435-image-registry-private-configuration\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.380667 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.380548    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84e4fef2-7c44-4c75-a75a-20fc52336320-cert\") pod \"ingress-canary-t8wnw\" (UID: \"84e4fef2-7c44-4c75-a75a-20fc52336320\") " pod="openshift-ingress-canary/ingress-canary-t8wnw"
Sep 16 12:01:15.386252 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.386222    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-bound-sa-token\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.386252 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.386236    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttmq5\" (UniqueName: \"kubernetes.io/projected/84e4fef2-7c44-4c75-a75a-20fc52336320-kube-api-access-ttmq5\") pod \"ingress-canary-t8wnw\" (UID: \"84e4fef2-7c44-4c75-a75a-20fc52336320\") " pod="openshift-ingress-canary/ingress-canary-t8wnw"
Sep 16 12:01:15.388278 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.388259    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ks5b\" (UniqueName: \"kubernetes.io/projected/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-kube-api-access-8ks5b\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:15.388388 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.388370    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jcwg\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-kube-api-access-8jcwg\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.445865 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.445832    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t8wnw"
Sep 16 12:01:15.781400 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.781325    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t8wnw"]
Sep 16 12:01:15.785286 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:01:15.785264    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e4fef2_7c44_4c75_a75a_20fc52336320.slice/crio-d4f22987941d7b7922f849bc5301994edd189b3facd772eb5f01dcc196c95c80 WatchSource:0}: Error finding container d4f22987941d7b7922f849bc5301994edd189b3facd772eb5f01dcc196c95c80: Status 404 returned error can't find the container with id d4f22987941d7b7922f849bc5301994edd189b3facd772eb5f01dcc196c95c80
Sep 16 12:01:15.881347 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.881304    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:15.881532 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:15.881365    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:15.881532 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:15.881474    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:01:15.881532 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:15.881494    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:01:15.881532 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:15.881517    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d44784f4-6mqnw: secret "image-registry-tls" not found
Sep 16 12:01:15.881532 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:15.881532    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls podName:aab5b1d1-1603-47a0-a5cc-0d136c7f678a nodeName:}" failed. No retries permitted until 2026-09-16 12:01:16.881516794 +0000 UTC m=+36.132054324 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls") pod "dns-default-wf5rh" (UID: "aab5b1d1-1603-47a0-a5cc-0d136c7f678a") : secret "dns-default-metrics-tls" not found
Sep 16 12:01:15.881763 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:15.881576    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls podName:751c95c1-cd17-48f4-9d07-8c7c6c312435 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:16.881557187 +0000 UTC m=+36.132094715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls") pod "image-registry-59d44784f4-6mqnw" (UID: "751c95c1-cd17-48f4-9d07-8c7c6c312435") : secret "image-registry-tls" not found
Sep 16 12:01:16.206489 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:16.206454    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:01:16.207196 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:16.206601    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:01:16.207196 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:16.206459    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:01:16.209031 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:16.209006    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:01:16.210407 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:16.210222    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:01:16.210407 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:16.210237    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rfjm7\""
Sep 16 12:01:16.210407 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:16.210273    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:01:16.210407 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:16.210305    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hp2p6\""
Sep 16 12:01:16.210640 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:16.210449    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:01:16.387547 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:16.387518    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t8wnw" event={"ID":"84e4fef2-7c44-4c75-a75a-20fc52336320","Type":"ContainerStarted","Data":"d4f22987941d7b7922f849bc5301994edd189b3facd772eb5f01dcc196c95c80"}
Sep 16 12:01:16.890909 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:16.890875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:16.891095 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:16.890927    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:16.891095 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:16.891049    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:01:16.891095 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:16.891073    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:01:16.891095 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:16.891075    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d44784f4-6mqnw: secret "image-registry-tls" not found
Sep 16 12:01:16.891260 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:16.891184    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls podName:aab5b1d1-1603-47a0-a5cc-0d136c7f678a nodeName:}" failed. No retries permitted until 2026-09-16 12:01:18.89113803 +0000 UTC m=+38.141675578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls") pod "dns-default-wf5rh" (UID: "aab5b1d1-1603-47a0-a5cc-0d136c7f678a") : secret "dns-default-metrics-tls" not found
Sep 16 12:01:16.891260 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:16.891208    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls podName:751c95c1-cd17-48f4-9d07-8c7c6c312435 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:18.891196293 +0000 UTC m=+38.141733855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls") pod "image-registry-59d44784f4-6mqnw" (UID: "751c95c1-cd17-48f4-9d07-8c7c6c312435") : secret "image-registry-tls" not found
Sep 16 12:01:18.910182 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:18.910142    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:18.910749 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:18.910195    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:18.910749 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:18.910311    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:01:18.910749 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:18.910315    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:01:18.910749 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:18.910332    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d44784f4-6mqnw: secret "image-registry-tls" not found
Sep 16 12:01:18.910749 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:18.910380    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls podName:751c95c1-cd17-48f4-9d07-8c7c6c312435 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:22.910361112 +0000 UTC m=+42.160898654 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls") pod "image-registry-59d44784f4-6mqnw" (UID: "751c95c1-cd17-48f4-9d07-8c7c6c312435") : secret "image-registry-tls" not found
Sep 16 12:01:18.910749 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:18.910397    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls podName:aab5b1d1-1603-47a0-a5cc-0d136c7f678a nodeName:}" failed. No retries permitted until 2026-09-16 12:01:22.910388432 +0000 UTC m=+42.160925958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls") pod "dns-default-wf5rh" (UID: "aab5b1d1-1603-47a0-a5cc-0d136c7f678a") : secret "dns-default-metrics-tls" not found
Sep 16 12:01:20.399645 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:20.399370    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t8wnw" event={"ID":"84e4fef2-7c44-4c75-a75a-20fc52336320","Type":"ContainerStarted","Data":"989a5069f649509a7a50b54508ec14d5d5f7dbb13e859aa94eb9744b31543e3a"}
Sep 16 12:01:20.402893 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:20.402862    2582 generic.go:356] "Generic (PLEG): container finished" podID="25085fb2-c567-4b20-bd8d-6241d7cce1cf" containerID="97c6926f757faa8e34061c460d2daa133eaf6d6731ad2ed82ae0c19b286e6b13" exitCode=0
Sep 16 12:01:20.403049 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:20.402906    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w69gx" event={"ID":"25085fb2-c567-4b20-bd8d-6241d7cce1cf","Type":"ContainerDied","Data":"97c6926f757faa8e34061c460d2daa133eaf6d6731ad2ed82ae0c19b286e6b13"}
Sep 16 12:01:20.415839 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:20.415154    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-t8wnw" podStartSLOduration=1.468742843 podStartE2EDuration="5.415135318s" totalImagesPullingTime="3.946392475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:01:20.414429966 +0000 UTC m=+39.664967515" watchObservedRunningTime="2026-09-16 12:01:20.415135318 +0000 UTC m=+39.665672844"
Sep 16 12:01:21.408888 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:21.408852    2582 generic.go:356] "Generic (PLEG): container finished" podID="25085fb2-c567-4b20-bd8d-6241d7cce1cf" containerID="2a162d740b0867e96fcb4be927e31fbd91ea923ce9c493b16f1dd84058f1bb51" exitCode=0
Sep 16 12:01:21.409283 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:21.408942    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w69gx" event={"ID":"25085fb2-c567-4b20-bd8d-6241d7cce1cf","Type":"ContainerDied","Data":"2a162d740b0867e96fcb4be927e31fbd91ea923ce9c493b16f1dd84058f1bb51"}
Sep 16 12:01:22.415502 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:22.415468    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w69gx" event={"ID":"25085fb2-c567-4b20-bd8d-6241d7cce1cf","Type":"ContainerStarted","Data":"2552bfa65e5e8504f42c34160d5d4680152316843e36f9a3a2e8231b5f4c13e1"}
Sep 16 12:01:22.439978 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:22.439931    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w69gx" podStartSLOduration=9.353781282 podStartE2EDuration="41.439914732s" totalImagesPullingTime="32.08613345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:00:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:01:22.437696772 +0000 UTC m=+41.688234338" watchObservedRunningTime="2026-09-16 12:01:22.439914732 +0000 UTC m=+41.690452280"
Sep 16 12:01:22.942415 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:22.942370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:22.942415 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:22.942417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:22.942631 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:22.942547    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:01:22.942631 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:22.942552    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:01:22.942631 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:22.942572    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d44784f4-6mqnw: secret "image-registry-tls" not found
Sep 16 12:01:22.942631 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:22.942600    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls podName:aab5b1d1-1603-47a0-a5cc-0d136c7f678a nodeName:}" failed. No retries permitted until 2026-09-16 12:01:30.942587598 +0000 UTC m=+50.193125125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls") pod "dns-default-wf5rh" (UID: "aab5b1d1-1603-47a0-a5cc-0d136c7f678a") : secret "dns-default-metrics-tls" not found
Sep 16 12:01:22.942631 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:22.942619    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls podName:751c95c1-cd17-48f4-9d07-8c7c6c312435 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:30.942605151 +0000 UTC m=+50.193142677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls") pod "image-registry-59d44784f4-6mqnw" (UID: "751c95c1-cd17-48f4-9d07-8c7c6c312435") : secret "image-registry-tls" not found
Sep 16 12:01:31.001401 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:31.001351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:31.001881 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:31.001409    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:31.001881 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:31.001521    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:01:31.001881 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:31.001543    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d44784f4-6mqnw: secret "image-registry-tls" not found
Sep 16 12:01:31.001881 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:31.001592    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls podName:751c95c1-cd17-48f4-9d07-8c7c6c312435 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:47.001577921 +0000 UTC m=+66.252115447 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls") pod "image-registry-59d44784f4-6mqnw" (UID: "751c95c1-cd17-48f4-9d07-8c7c6c312435") : secret "image-registry-tls" not found
Sep 16 12:01:31.001881 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:31.001524    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:01:31.001881 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:31.001645    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls podName:aab5b1d1-1603-47a0-a5cc-0d136c7f678a nodeName:}" failed. No retries permitted until 2026-09-16 12:01:47.001635005 +0000 UTC m=+66.252172545 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls") pod "dns-default-wf5rh" (UID: "aab5b1d1-1603-47a0-a5cc-0d136c7f678a") : secret "dns-default-metrics-tls" not found
Sep 16 12:01:42.384884 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:42.384853    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcx6d"
Sep 16 12:01:47.022039 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:47.021993    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs\") pod \"network-metrics-daemon-bflx7\" (UID: \"9a90e5c3-5fcb-4016-8682-3803c3121620\") " pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:01:47.022486 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:47.022075    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:01:47.022486 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:47.022103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret\") pod \"global-pull-secret-syncer-8drdm\" (UID: \"bef09744-222b-4d55-9d87-aca589bd4f13\") " pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:01:47.022486 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:47.022127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:01:47.022486 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:47.022240    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:01:47.022486 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:47.022243    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:01:47.022486 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:47.022270    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d44784f4-6mqnw: secret "image-registry-tls" not found
Sep 16 12:01:47.022486 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:47.022308    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls podName:aab5b1d1-1603-47a0-a5cc-0d136c7f678a nodeName:}" failed. No retries permitted until 2026-09-16 12:02:19.022288574 +0000 UTC m=+98.272826115 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls") pod "dns-default-wf5rh" (UID: "aab5b1d1-1603-47a0-a5cc-0d136c7f678a") : secret "dns-default-metrics-tls" not found
Sep 16 12:01:47.022486 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:47.022327    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls podName:751c95c1-cd17-48f4-9d07-8c7c6c312435 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:19.022314109 +0000 UTC m=+98.272851636 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls") pod "image-registry-59d44784f4-6mqnw" (UID: "751c95c1-cd17-48f4-9d07-8c7c6c312435") : secret "image-registry-tls" not found
Sep 16 12:01:47.025273 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:47.025256    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:01:47.025328 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:47.025300    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:01:47.033133 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:47.033110    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:01:47.033257 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:01:47.033155    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs podName:9a90e5c3-5fcb-4016-8682-3803c3121620 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:51.033143152 +0000 UTC m=+130.283680693 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs") pod "network-metrics-daemon-bflx7" (UID: "9a90e5c3-5fcb-4016-8682-3803c3121620") : secret "metrics-daemon-secret" not found
Sep 16 12:01:47.036139 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:47.036118    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef09744-222b-4d55-9d87-aca589bd4f13-original-pull-secret\") pod \"global-pull-secret-syncer-8drdm\" (UID: \"bef09744-222b-4d55-9d87-aca589bd4f13\") " pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:01:47.122941 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:47.122903    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzrkw\" (UniqueName: \"kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw\") pod \"network-check-target-2ps6l\" (UID: \"becc876e-d896-4811-9ee1-0be40fe69dda\") " pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:01:47.125851 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:47.125831    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:01:47.132569 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:47.132555    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8drdm"
Sep 16 12:01:47.136224 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:47.136209    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:01:47.146338 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:47.146316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzrkw\" (UniqueName: \"kubernetes.io/projected/becc876e-d896-4811-9ee1-0be40fe69dda-kube-api-access-lzrkw\") pod \"network-check-target-2ps6l\" (UID: \"becc876e-d896-4811-9ee1-0be40fe69dda\") " pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:01:47.405098 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:47.405067    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8drdm"]
Sep 16 12:01:47.408649 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:01:47.408620    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef09744_222b_4d55_9d87_aca589bd4f13.slice/crio-6b7c51e572aef30af0a6d9893af31a300e6bf54eb8fd808a0b1d84bcc617ff3c WatchSource:0}: Error finding container 6b7c51e572aef30af0a6d9893af31a300e6bf54eb8fd808a0b1d84bcc617ff3c: Status 404 returned error can't find the container with id 6b7c51e572aef30af0a6d9893af31a300e6bf54eb8fd808a0b1d84bcc617ff3c
Sep 16 12:01:47.421862 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:47.421842    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rfjm7\""
Sep 16 12:01:47.430142 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:47.430123    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:01:47.480600 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:47.480548    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8drdm" event={"ID":"bef09744-222b-4d55-9d87-aca589bd4f13","Type":"ContainerStarted","Data":"6b7c51e572aef30af0a6d9893af31a300e6bf54eb8fd808a0b1d84bcc617ff3c"}
Sep 16 12:01:47.701533 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:47.700540    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2ps6l"]
Sep 16 12:01:47.704951 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:01:47.704923    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbecc876e_d896_4811_9ee1_0be40fe69dda.slice/crio-d4b12fca5be22df3fc8c29d50d62d0b9cbe7f03bd5928246975b0988f02c439c WatchSource:0}: Error finding container d4b12fca5be22df3fc8c29d50d62d0b9cbe7f03bd5928246975b0988f02c439c: Status 404 returned error can't find the container with id d4b12fca5be22df3fc8c29d50d62d0b9cbe7f03bd5928246975b0988f02c439c
Sep 16 12:01:48.485157 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:48.485122    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2ps6l" event={"ID":"becc876e-d896-4811-9ee1-0be40fe69dda","Type":"ContainerStarted","Data":"d4b12fca5be22df3fc8c29d50d62d0b9cbe7f03bd5928246975b0988f02c439c"}
Sep 16 12:01:53.503134 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:53.503094    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8drdm" event={"ID":"bef09744-222b-4d55-9d87-aca589bd4f13","Type":"ContainerStarted","Data":"d735eb95439851f1bebb700306c0811631b1e4ab8601af7ee29587cff7da738d"}
Sep 16 12:01:53.504748 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:53.504712    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2ps6l" event={"ID":"becc876e-d896-4811-9ee1-0be40fe69dda","Type":"ContainerStarted","Data":"e6e46dec09773d23165587c78a263cc151c171c0e9c77629e35fca756e56b1c1"}
Sep 16 12:01:53.504856 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:53.504840    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:01:53.519465 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:53.519423    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8drdm" podStartSLOduration=67.241421248 podStartE2EDuration="1m12.519408704s" totalImagesPullingTime="5.277987456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:01:53.518846352 +0000 UTC m=+72.769383901" watchObservedRunningTime="2026-09-16 12:01:53.519408704 +0000 UTC m=+72.769946318"
Sep 16 12:01:53.535561 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:01:53.535521    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2ps6l" podStartSLOduration=67.565182138 podStartE2EDuration="1m12.535505043s" totalImagesPullingTime="4.970322905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:01:53.534950281 +0000 UTC m=+72.785487830" watchObservedRunningTime="2026-09-16 12:01:53.535505043 +0000 UTC m=+72.786042591"
Sep 16 12:02:17.697238 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:17.697194    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"]
Sep 16 12:02:17.701510 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:17.701487    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"
Sep 16 12:02:17.705720 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:17.705699    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 12:02:17.705873 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:17.705764    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 12:02:17.705873 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:17.705701    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 12:02:17.705873 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:17.705701    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 12:02:17.712788 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:17.712705    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"]
Sep 16 12:02:17.845718 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:17.845678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89afa634-bb70-47eb-827d-66ac4c054e13-tmp\") pod \"klusterlet-addon-workmgr-8468c4574f-xhslx\" (UID: \"89afa634-bb70-47eb-827d-66ac4c054e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"
Sep 16 12:02:17.845911 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:17.845755    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/89afa634-bb70-47eb-827d-66ac4c054e13-klusterlet-config\") pod \"klusterlet-addon-workmgr-8468c4574f-xhslx\" (UID: \"89afa634-bb70-47eb-827d-66ac4c054e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"
Sep 16 12:02:17.845911 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:17.845777    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sllwl\" (UniqueName: \"kubernetes.io/projected/89afa634-bb70-47eb-827d-66ac4c054e13-kube-api-access-sllwl\") pod \"klusterlet-addon-workmgr-8468c4574f-xhslx\" (UID: \"89afa634-bb70-47eb-827d-66ac4c054e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"
Sep 16 12:02:17.946244 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:17.946214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89afa634-bb70-47eb-827d-66ac4c054e13-tmp\") pod \"klusterlet-addon-workmgr-8468c4574f-xhslx\" (UID: \"89afa634-bb70-47eb-827d-66ac4c054e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"
Sep 16 12:02:17.946366 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:17.946265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/89afa634-bb70-47eb-827d-66ac4c054e13-klusterlet-config\") pod \"klusterlet-addon-workmgr-8468c4574f-xhslx\" (UID: \"89afa634-bb70-47eb-827d-66ac4c054e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"
Sep 16 12:02:17.946366 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:17.946282    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sllwl\" (UniqueName: \"kubernetes.io/projected/89afa634-bb70-47eb-827d-66ac4c054e13-kube-api-access-sllwl\") pod \"klusterlet-addon-workmgr-8468c4574f-xhslx\" (UID: \"89afa634-bb70-47eb-827d-66ac4c054e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"
Sep 16 12:02:17.946659 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:17.946632    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89afa634-bb70-47eb-827d-66ac4c054e13-tmp\") pod \"klusterlet-addon-workmgr-8468c4574f-xhslx\" (UID: \"89afa634-bb70-47eb-827d-66ac4c054e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"
Sep 16 12:02:17.948970 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:17.948929    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/89afa634-bb70-47eb-827d-66ac4c054e13-klusterlet-config\") pod \"klusterlet-addon-workmgr-8468c4574f-xhslx\" (UID: \"89afa634-bb70-47eb-827d-66ac4c054e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"
Sep 16 12:02:17.954852 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:17.954830    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sllwl\" (UniqueName: \"kubernetes.io/projected/89afa634-bb70-47eb-827d-66ac4c054e13-kube-api-access-sllwl\") pod \"klusterlet-addon-workmgr-8468c4574f-xhslx\" (UID: \"89afa634-bb70-47eb-827d-66ac4c054e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"
Sep 16 12:02:18.009839 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:18.009804    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"
Sep 16 12:02:18.277821 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:18.277586    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"]
Sep 16 12:02:18.280441 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:02:18.280416    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89afa634_bb70_47eb_827d_66ac4c054e13.slice/crio-9d1b6b70abe55dcd15f407050f04b0941b15cd52e4088275e64928fa97108c69 WatchSource:0}: Error finding container 9d1b6b70abe55dcd15f407050f04b0941b15cd52e4088275e64928fa97108c69: Status 404 returned error can't find the container with id 9d1b6b70abe55dcd15f407050f04b0941b15cd52e4088275e64928fa97108c69
Sep 16 12:02:18.567046 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:18.566954    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx" event={"ID":"89afa634-bb70-47eb-827d-66ac4c054e13","Type":"ContainerStarted","Data":"9d1b6b70abe55dcd15f407050f04b0941b15cd52e4088275e64928fa97108c69"}
Sep 16 12:02:19.055234 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:19.055198    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:02:19.055234 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:19.055235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:02:19.055711 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:02:19.055370    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:02:19.055711 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:02:19.055392    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:02:19.055711 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:02:19.055412    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d44784f4-6mqnw: secret "image-registry-tls" not found
Sep 16 12:02:19.055711 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:02:19.055431    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls podName:aab5b1d1-1603-47a0-a5cc-0d136c7f678a nodeName:}" failed. No retries permitted until 2026-09-16 12:03:23.055418027 +0000 UTC m=+162.305955554 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls") pod "dns-default-wf5rh" (UID: "aab5b1d1-1603-47a0-a5cc-0d136c7f678a") : secret "dns-default-metrics-tls" not found
Sep 16 12:02:19.055711 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:02:19.055465    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls podName:751c95c1-cd17-48f4-9d07-8c7c6c312435 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:23.055447732 +0000 UTC m=+162.305985274 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls") pod "image-registry-59d44784f4-6mqnw" (UID: "751c95c1-cd17-48f4-9d07-8c7c6c312435") : secret "image-registry-tls" not found
Sep 16 12:02:23.582439 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:23.582409    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx" event={"ID":"89afa634-bb70-47eb-827d-66ac4c054e13","Type":"ContainerStarted","Data":"9a61d847fe7df7b649208ffadb1f6a83ab83e77ef4471889de5da11abea158dd"}
Sep 16 12:02:23.582887 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:23.582658    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"
Sep 16 12:02:23.584221 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:23.584200    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"
Sep 16 12:02:23.599932 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:23.599885    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx" podStartSLOduration=1.585056336 podStartE2EDuration="6.599870208s" totalImagesPullingTime="5.014813872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:23.598838225 +0000 UTC m=+102.849375773" watchObservedRunningTime="2026-09-16 12:02:23.599870208 +0000 UTC m=+102.850407756"
Sep 16 12:02:24.510153 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:24.510124    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2ps6l"
Sep 16 12:02:51.086969 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:02:51.086912    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs\") pod \"network-metrics-daemon-bflx7\" (UID: \"9a90e5c3-5fcb-4016-8682-3803c3121620\") " pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:02:51.087466 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:02:51.087043    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:02:51.087466 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:02:51.087102    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs podName:9a90e5c3-5fcb-4016-8682-3803c3121620 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:53.087088279 +0000 UTC m=+252.337625806 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs") pod "network-metrics-daemon-bflx7" (UID: "9a90e5c3-5fcb-4016-8682-3803c3121620") : secret "metrics-daemon-secret" not found
Sep 16 12:03:18.132194 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:03:18.132147    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-59d44784f4-6mqnw" podUID="751c95c1-cd17-48f4-9d07-8c7c6c312435"
Sep 16 12:03:18.164478 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:03:18.164448    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wf5rh" podUID="aab5b1d1-1603-47a0-a5cc-0d136c7f678a"
Sep 16 12:03:18.718880 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:03:18.718851    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:03:19.226407 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:03:19.226354    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:03:23.115471 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:03:23.115425    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:03:23.115471 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:03:23.115468    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:03:23.115963 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:03:23.115587    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:03:23.115963 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:03:23.115599    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:03:23.115963 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:03:23.115609    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59d44784f4-6mqnw: secret "image-registry-tls" not found
Sep 16 12:03:23.115963 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:03:23.115671    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls podName:751c95c1-cd17-48f4-9d07-8c7c6c312435 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:25.115654875 +0000 UTC m=+284.366192402 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls") pod "image-registry-59d44784f4-6mqnw" (UID: "751c95c1-cd17-48f4-9d07-8c7c6c312435") : secret "image-registry-tls" not found
Sep 16 12:03:23.115963 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:03:23.115685    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls podName:aab5b1d1-1603-47a0-a5cc-0d136c7f678a nodeName:}" failed. No retries permitted until 2026-09-16 12:05:25.115679123 +0000 UTC m=+284.366216650 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls") pod "dns-default-wf5rh" (UID: "aab5b1d1-1603-47a0-a5cc-0d136c7f678a") : secret "dns-default-metrics-tls" not found
Sep 16 12:03:32.206473 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:03:32.206388    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:03:33.206562 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:03:33.206525    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:04:23.583776 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:04:23.583698    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx" podUID="89afa634-bb70-47eb-827d-66ac4c054e13" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.9:8000/readyz\": dial tcp 10.133.0.9:8000: connect: connection refused"
Sep 16 12:04:23.880988 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:04:23.880960    2582 generic.go:356] "Generic (PLEG): container finished" podID="89afa634-bb70-47eb-827d-66ac4c054e13" containerID="9a61d847fe7df7b649208ffadb1f6a83ab83e77ef4471889de5da11abea158dd" exitCode=1
Sep 16 12:04:23.881149 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:04:23.881000    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx" event={"ID":"89afa634-bb70-47eb-827d-66ac4c054e13","Type":"ContainerDied","Data":"9a61d847fe7df7b649208ffadb1f6a83ab83e77ef4471889de5da11abea158dd"}
Sep 16 12:04:23.881345 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:04:23.881329    2582 scope.go:117] "RemoveContainer" containerID="9a61d847fe7df7b649208ffadb1f6a83ab83e77ef4471889de5da11abea158dd"
Sep 16 12:04:24.885431 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:04:24.885398    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx" event={"ID":"89afa634-bb70-47eb-827d-66ac4c054e13","Type":"ContainerStarted","Data":"3444a8fec699bbfb2b01438e07a5eb49d2a23b701ad0d266973fdda103572b5a"}
Sep 16 12:04:24.885897 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:04:24.885724    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"
Sep 16 12:04:24.886452 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:04:24.886434    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8468c4574f-xhslx"
Sep 16 12:04:53.167853 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:04:53.167785    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs\") pod \"network-metrics-daemon-bflx7\" (UID: \"9a90e5c3-5fcb-4016-8682-3803c3121620\") " pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:04:53.168340 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:04:53.167945    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:04:53.168340 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:04:53.168015    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs podName:9a90e5c3-5fcb-4016-8682-3803c3121620 nodeName:}" failed. No retries permitted until 2026-09-16 12:06:55.168000693 +0000 UTC m=+374.418538219 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs") pod "network-metrics-daemon-bflx7" (UID: "9a90e5c3-5fcb-4016-8682-3803c3121620") : secret "metrics-daemon-secret" not found
Sep 16 12:05:06.333004 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.332920    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-9hpds"]
Sep 16 12:05:06.335650 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.335632    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-69bbc96588-pkg66"]
Sep 16 12:05:06.335808 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.335790    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.338331 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.338313    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:06.339254 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.339228    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 12:05:06.339254 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.339243    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 12:05:06.339400 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.339250    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 12:05:06.339916 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.339901    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-g7l6g\""
Sep 16 12:05:06.340175 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.340159    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 12:05:06.342270 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.342113    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 12:05:06.342434 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.342372    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 12:05:06.342510 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.342468    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 12:05:06.342564 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.342528    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 12:05:06.342725 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.342699    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 12:05:06.343219 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.343202    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 12:05:06.343319 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.343276    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-v7rrj\""
Sep 16 12:05:06.348396 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.348379    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 12:05:06.349488 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.349469    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-24dfb"]
Sep 16 12:05:06.352409 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.352391    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-24dfb"
Sep 16 12:05:06.353161 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.353141    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-9hpds"]
Sep 16 12:05:06.355032 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.355013    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 12:05:06.355124 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.355020    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:05:06.355124 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.355020    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-d5rz9\""
Sep 16 12:05:06.365475 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.365453    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-24dfb"]
Sep 16 12:05:06.368076 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.368038    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/747ed2fa-e045-4333-a75c-20d76a36ecf6-service-ca-bundle\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.368176 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.368118    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:06.368176 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.368153    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-default-certificate\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:06.368269 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.368179    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkztj\" (UniqueName: \"kubernetes.io/projected/82d9505e-b799-4990-9927-e308c84e5292-kube-api-access-kkztj\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:06.368269 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.368223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/747ed2fa-e045-4333-a75c-20d76a36ecf6-tmp\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.368269 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.368243    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n76n7\" (UniqueName: \"kubernetes.io/projected/747ed2fa-e045-4333-a75c-20d76a36ecf6-kube-api-access-n76n7\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.368495 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.368286    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/747ed2fa-e045-4333-a75c-20d76a36ecf6-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.368495 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.368322    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/747ed2fa-e045-4333-a75c-20d76a36ecf6-snapshots\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.368495 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.368365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:06.368495 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.368413    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/747ed2fa-e045-4333-a75c-20d76a36ecf6-serving-cert\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.368495 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.368464    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-stats-auth\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:06.373109 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.373084    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69bbc96588-pkg66"]
Sep 16 12:05:06.416912 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.416873    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-7n9hs"]
Sep 16 12:05:06.419937 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.419911    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7n9hs"
Sep 16 12:05:06.422440 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.422420    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ntgjq\""
Sep 16 12:05:06.429533 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.429512    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-7n9hs"]
Sep 16 12:05:06.469754 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.469700    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/747ed2fa-e045-4333-a75c-20d76a36ecf6-service-ca-bundle\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.469754 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.469753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsqch\" (UniqueName: \"kubernetes.io/projected/b7b80a25-50dd-45f1-8118-23adabe85425-kube-api-access-lsqch\") pod \"volume-data-source-validator-86695b7cc7-24dfb\" (UID: \"b7b80a25-50dd-45f1-8118-23adabe85425\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-24dfb"
Sep 16 12:05:06.469949 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.469869    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-default-certificate\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:06.469949 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.469897    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkztj\" (UniqueName: \"kubernetes.io/projected/82d9505e-b799-4990-9927-e308c84e5292-kube-api-access-kkztj\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:06.469949 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.469917    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2ksc\" (UniqueName: \"kubernetes.io/projected/c2a45b6e-16e8-44cf-9a10-92ba199ce3cd-kube-api-access-w2ksc\") pod \"network-check-source-6f8b768455-7n9hs\" (UID: \"c2a45b6e-16e8-44cf-9a10-92ba199ce3cd\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-7n9hs"
Sep 16 12:05:06.469949 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.469942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/747ed2fa-e045-4333-a75c-20d76a36ecf6-tmp\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.470171 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.469968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n76n7\" (UniqueName: \"kubernetes.io/projected/747ed2fa-e045-4333-a75c-20d76a36ecf6-kube-api-access-n76n7\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.470171 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.469993    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/747ed2fa-e045-4333-a75c-20d76a36ecf6-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.470171 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.470021    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:06.470171 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.470049    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/747ed2fa-e045-4333-a75c-20d76a36ecf6-serving-cert\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.470171 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.470081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-stats-auth\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:06.470171 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.470126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:06.470171 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.470162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/747ed2fa-e045-4333-a75c-20d76a36ecf6-snapshots\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.470531 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:06.470256    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle podName:82d9505e-b799-4990-9927-e308c84e5292 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:06.970230957 +0000 UTC m=+266.220768502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle") pod "router-default-69bbc96588-pkg66" (UID: "82d9505e-b799-4990-9927-e308c84e5292") : configmap references non-existent config key: service-ca.crt
Sep 16 12:05:06.470531 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:06.470326    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:05:06.470531 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:06.470379    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs podName:82d9505e-b799-4990-9927-e308c84e5292 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:06.970365682 +0000 UTC m=+266.220903212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs") pod "router-default-69bbc96588-pkg66" (UID: "82d9505e-b799-4990-9927-e308c84e5292") : secret "router-metrics-certs-default" not found
Sep 16 12:05:06.470531 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.470396    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/747ed2fa-e045-4333-a75c-20d76a36ecf6-tmp\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.470531 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.470399    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/747ed2fa-e045-4333-a75c-20d76a36ecf6-service-ca-bundle\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.470808 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.470728    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/747ed2fa-e045-4333-a75c-20d76a36ecf6-snapshots\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.470948 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.470932    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/747ed2fa-e045-4333-a75c-20d76a36ecf6-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.472699 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.472677    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/747ed2fa-e045-4333-a75c-20d76a36ecf6-serving-cert\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.473138 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.473119    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-stats-auth\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:06.473215 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.473196    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-default-certificate\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:06.481013 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.480990    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkztj\" (UniqueName: \"kubernetes.io/projected/82d9505e-b799-4990-9927-e308c84e5292-kube-api-access-kkztj\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:06.481113 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.481097    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n76n7\" (UniqueName: \"kubernetes.io/projected/747ed2fa-e045-4333-a75c-20d76a36ecf6-kube-api-access-n76n7\") pod \"insights-operator-5844b856db-9hpds\" (UID: \"747ed2fa-e045-4333-a75c-20d76a36ecf6\") " pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.570533 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.570482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsqch\" (UniqueName: \"kubernetes.io/projected/b7b80a25-50dd-45f1-8118-23adabe85425-kube-api-access-lsqch\") pod \"volume-data-source-validator-86695b7cc7-24dfb\" (UID: \"b7b80a25-50dd-45f1-8118-23adabe85425\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-24dfb"
Sep 16 12:05:06.570533 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.570552    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2ksc\" (UniqueName: \"kubernetes.io/projected/c2a45b6e-16e8-44cf-9a10-92ba199ce3cd-kube-api-access-w2ksc\") pod \"network-check-source-6f8b768455-7n9hs\" (UID: \"c2a45b6e-16e8-44cf-9a10-92ba199ce3cd\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-7n9hs"
Sep 16 12:05:06.579965 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.579939    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsqch\" (UniqueName: \"kubernetes.io/projected/b7b80a25-50dd-45f1-8118-23adabe85425-kube-api-access-lsqch\") pod \"volume-data-source-validator-86695b7cc7-24dfb\" (UID: \"b7b80a25-50dd-45f1-8118-23adabe85425\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-24dfb"
Sep 16 12:05:06.580131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.580110    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2ksc\" (UniqueName: \"kubernetes.io/projected/c2a45b6e-16e8-44cf-9a10-92ba199ce3cd-kube-api-access-w2ksc\") pod \"network-check-source-6f8b768455-7n9hs\" (UID: \"c2a45b6e-16e8-44cf-9a10-92ba199ce3cd\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-7n9hs"
Sep 16 12:05:06.645246 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.645157    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-9hpds"
Sep 16 12:05:06.660535 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.660513    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-24dfb"
Sep 16 12:05:06.729089 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.729063    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7n9hs"
Sep 16 12:05:06.916006 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.915976    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-9hpds"]
Sep 16 12:05:06.918390 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:05:06.918365    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747ed2fa_e045_4333_a75c_20d76a36ecf6.slice/crio-b19fab7f452dd265b23368c28678d6c74c149d868f674a913910454749a2608b WatchSource:0}: Error finding container b19fab7f452dd265b23368c28678d6c74c149d868f674a913910454749a2608b: Status 404 returned error can't find the container with id b19fab7f452dd265b23368c28678d6c74c149d868f674a913910454749a2608b
Sep 16 12:05:06.938983 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.938959    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-24dfb"]
Sep 16 12:05:06.941237 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:05:06.941213    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b80a25_50dd_45f1_8118_23adabe85425.slice/crio-16f98402baef232edbbda118e270f3b0bcd0537db422a61e0cce84c08413784a WatchSource:0}: Error finding container 16f98402baef232edbbda118e270f3b0bcd0537db422a61e0cce84c08413784a: Status 404 returned error can't find the container with id 16f98402baef232edbbda118e270f3b0bcd0537db422a61e0cce84c08413784a
Sep 16 12:05:06.974165 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.974132    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:06.974294 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.974184    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:06.974294 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:06.974284    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:05:06.974355 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:06.974304    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle podName:82d9505e-b799-4990-9927-e308c84e5292 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:07.974283018 +0000 UTC m=+267.224820560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle") pod "router-default-69bbc96588-pkg66" (UID: "82d9505e-b799-4990-9927-e308c84e5292") : configmap references non-existent config key: service-ca.crt
Sep 16 12:05:06.974355 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:06.974330    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs podName:82d9505e-b799-4990-9927-e308c84e5292 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:07.974319905 +0000 UTC m=+267.224857434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs") pod "router-default-69bbc96588-pkg66" (UID: "82d9505e-b799-4990-9927-e308c84e5292") : secret "router-metrics-certs-default" not found
Sep 16 12:05:06.995531 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.995497    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9hpds" event={"ID":"747ed2fa-e045-4333-a75c-20d76a36ecf6","Type":"ContainerStarted","Data":"b19fab7f452dd265b23368c28678d6c74c149d868f674a913910454749a2608b"}
Sep 16 12:05:06.996538 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.996513    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-24dfb" event={"ID":"b7b80a25-50dd-45f1-8118-23adabe85425","Type":"ContainerStarted","Data":"16f98402baef232edbbda118e270f3b0bcd0537db422a61e0cce84c08413784a"}
Sep 16 12:05:06.999325 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:06.999301    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-7n9hs"]
Sep 16 12:05:07.002451 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:05:07.002429    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a45b6e_16e8_44cf_9a10_92ba199ce3cd.slice/crio-01bfc541467dd7472c5332cb463fcb14300afc097045d8ce624f52a10522e342 WatchSource:0}: Error finding container 01bfc541467dd7472c5332cb463fcb14300afc097045d8ce624f52a10522e342: Status 404 returned error can't find the container with id 01bfc541467dd7472c5332cb463fcb14300afc097045d8ce624f52a10522e342
Sep 16 12:05:07.983228 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:07.983173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:07.983711 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:07.983272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:07.983711 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:07.983364    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle podName:82d9505e-b799-4990-9927-e308c84e5292 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:09.983341561 +0000 UTC m=+269.233879093 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle") pod "router-default-69bbc96588-pkg66" (UID: "82d9505e-b799-4990-9927-e308c84e5292") : configmap references non-existent config key: service-ca.crt
Sep 16 12:05:07.983711 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:07.983426    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:05:07.983711 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:07.983478    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs podName:82d9505e-b799-4990-9927-e308c84e5292 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:09.983463015 +0000 UTC m=+269.234000547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs") pod "router-default-69bbc96588-pkg66" (UID: "82d9505e-b799-4990-9927-e308c84e5292") : secret "router-metrics-certs-default" not found
Sep 16 12:05:08.003971 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:08.003872    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7n9hs" event={"ID":"c2a45b6e-16e8-44cf-9a10-92ba199ce3cd","Type":"ContainerStarted","Data":"537b3affdc535b68b890660a7a27e0e2d0e94446efb394c7706ecb29fdf9eb27"}
Sep 16 12:05:08.003971 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:08.003934    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7n9hs" event={"ID":"c2a45b6e-16e8-44cf-9a10-92ba199ce3cd","Type":"ContainerStarted","Data":"01bfc541467dd7472c5332cb463fcb14300afc097045d8ce624f52a10522e342"}
Sep 16 12:05:08.018749 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:08.018669    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7n9hs" podStartSLOduration=2.01865191 podStartE2EDuration="2.01865191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:05:08.018383981 +0000 UTC m=+267.268921530" watchObservedRunningTime="2026-09-16 12:05:08.01865191 +0000 UTC m=+267.269189458"
Sep 16 12:05:09.010043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:09.010001    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-24dfb" event={"ID":"b7b80a25-50dd-45f1-8118-23adabe85425","Type":"ContainerStarted","Data":"51c4fee17cb7aebc36a31cd6a5decb59775e407984b0f5613624d9f9d2bbc617"}
Sep 16 12:05:09.027199 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:09.027144    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-24dfb" podStartSLOduration=1.531382579 podStartE2EDuration="3.027125403s" totalImagesPullingTime="1.495742824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:05:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:05:09.026442135 +0000 UTC m=+268.276979688" watchObservedRunningTime="2026-09-16 12:05:09.027125403 +0000 UTC m=+268.277662952"
Sep 16 12:05:10.002631 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.002582    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:10.002857 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.002670    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:10.002857 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:10.002802    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle podName:82d9505e-b799-4990-9927-e308c84e5292 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:14.002782601 +0000 UTC m=+273.253320129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle") pod "router-default-69bbc96588-pkg66" (UID: "82d9505e-b799-4990-9927-e308c84e5292") : configmap references non-existent config key: service-ca.crt
Sep 16 12:05:10.002966 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:10.002860    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:05:10.002966 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:10.002913    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs podName:82d9505e-b799-4990-9927-e308c84e5292 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:14.002900641 +0000 UTC m=+273.253438171 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs") pod "router-default-69bbc96588-pkg66" (UID: "82d9505e-b799-4990-9927-e308c84e5292") : secret "router-metrics-certs-default" not found
Sep 16 12:05:10.014979 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.014950    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9hpds" event={"ID":"747ed2fa-e045-4333-a75c-20d76a36ecf6","Type":"ContainerStarted","Data":"9541acb19e45c2aa408bd3de854a912018f6a2d0b1509efbadc41faebc6af047"}
Sep 16 12:05:10.032549 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.032489    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-9hpds" podStartSLOduration=1.8243797210000001 podStartE2EDuration="4.032472849s" totalImagesPullingTime="2.208093128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:05:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:05:10.031166207 +0000 UTC m=+269.281703769" watchObservedRunningTime="2026-09-16 12:05:10.032472849 +0000 UTC m=+269.283010398"
Sep 16 12:05:10.195660 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.195612    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-57ksj"]
Sep 16 12:05:10.199378 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.199352    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:10.202621 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.202592    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 12:05:10.202621 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.202611    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lnmtw\""
Sep 16 12:05:10.202844 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.202644    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 12:05:10.211034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.211007    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-57ksj"]
Sep 16 12:05:10.305386 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.305286    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:10.305386 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.305354    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:10.305590 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.305423    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvh86\" (UniqueName: \"kubernetes.io/projected/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-api-access-xvh86\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:10.305590 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.305455    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-data-volume\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:10.305590 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.305539    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-crio-socket\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:10.406795 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.406754    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:10.406977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.406810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvh86\" (UniqueName: \"kubernetes.io/projected/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-api-access-xvh86\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:10.406977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.406830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-data-volume\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:10.407062 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:10.406986    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:05:10.407097 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:10.407066    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm podName:59d5b11b-1cb0-411d-bc9a-b16ff5a304c2 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:10.90704466 +0000 UTC m=+270.157582190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-57ksj" (UID: "59d5b11b-1cb0-411d-bc9a-b16ff5a304c2") : configmap "kube-rbac-proxy" not found
Sep 16 12:05:10.407160 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.407143    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-data-volume\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:10.407199 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.407150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-crio-socket\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:10.407240 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.407201    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-crio-socket\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:10.407240 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.407221    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:10.407336 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:10.407325    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:05:10.407373 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:10.407367    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls podName:59d5b11b-1cb0-411d-bc9a-b16ff5a304c2 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:10.907356128 +0000 UTC m=+270.157893655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-57ksj" (UID: "59d5b11b-1cb0-411d-bc9a-b16ff5a304c2") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:05:10.415874 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.415851    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvh86\" (UniqueName: \"kubernetes.io/projected/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-api-access-xvh86\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:10.912244 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.912206    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:10.912426 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:10.912261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:10.912426 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:10.912335    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:05:10.912426 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:10.912364    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:05:10.912426 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:10.912388    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm podName:59d5b11b-1cb0-411d-bc9a-b16ff5a304c2 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:11.912374753 +0000 UTC m=+271.162912280 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-57ksj" (UID: "59d5b11b-1cb0-411d-bc9a-b16ff5a304c2") : configmap "kube-rbac-proxy" not found
Sep 16 12:05:10.912558 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:10.912436    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls podName:59d5b11b-1cb0-411d-bc9a-b16ff5a304c2 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:11.912421837 +0000 UTC m=+271.162959379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-57ksj" (UID: "59d5b11b-1cb0-411d-bc9a-b16ff5a304c2") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:05:11.921354 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:11.921315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:11.921903 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:11.921372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:11.921903 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:11.921472    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:05:11.921903 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:11.921477    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:05:11.921903 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:11.921525    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm podName:59d5b11b-1cb0-411d-bc9a-b16ff5a304c2 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:13.921512117 +0000 UTC m=+273.172049643 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-57ksj" (UID: "59d5b11b-1cb0-411d-bc9a-b16ff5a304c2") : configmap "kube-rbac-proxy" not found
Sep 16 12:05:11.921903 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:11.921544    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls podName:59d5b11b-1cb0-411d-bc9a-b16ff5a304c2 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:13.921530105 +0000 UTC m=+273.172067631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-57ksj" (UID: "59d5b11b-1cb0-411d-bc9a-b16ff5a304c2") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:05:11.954563 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:11.954540    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4hzq6_e5308201-4e49-461a-be86-4c661c0d2783/dns-node-resolver/0.log"
Sep 16 12:05:12.953523 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:12.953495    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7q8hz_55b45601-95e3-4427-af82-2358088e7e8a/node-ca/0.log"
Sep 16 12:05:13.936466 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:13.936416    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:13.936678 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:13.936512    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:13.936678 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:13.936581    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:05:13.936678 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:13.936639    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:05:13.936867 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:13.936647    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls podName:59d5b11b-1cb0-411d-bc9a-b16ff5a304c2 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:17.936631005 +0000 UTC m=+277.187168534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-57ksj" (UID: "59d5b11b-1cb0-411d-bc9a-b16ff5a304c2") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:05:13.936867 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:13.936748    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm podName:59d5b11b-1cb0-411d-bc9a-b16ff5a304c2 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:17.936712285 +0000 UTC m=+277.187249816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-57ksj" (UID: "59d5b11b-1cb0-411d-bc9a-b16ff5a304c2") : configmap "kube-rbac-proxy" not found
Sep 16 12:05:13.953371 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:13.953340    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t8wnw_84e4fef2-7c44-4c75-a75a-20fc52336320/serve-healthcheck-canary/0.log"
Sep 16 12:05:14.037040 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:14.037013    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:14.037457 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:14.037090    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:14.037457 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:14.037168    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:05:14.037457 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:14.037213    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle podName:82d9505e-b799-4990-9927-e308c84e5292 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:22.037199437 +0000 UTC m=+281.287736964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle") pod "router-default-69bbc96588-pkg66" (UID: "82d9505e-b799-4990-9927-e308c84e5292") : configmap references non-existent config key: service-ca.crt
Sep 16 12:05:14.037457 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:14.037234    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs podName:82d9505e-b799-4990-9927-e308c84e5292 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:22.037221102 +0000 UTC m=+281.287758633 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs") pod "router-default-69bbc96588-pkg66" (UID: "82d9505e-b799-4990-9927-e308c84e5292") : secret "router-metrics-certs-default" not found
Sep 16 12:05:17.966344 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:17.966304    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:17.966771 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:17.966360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:17.966771 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:17.966443    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:05:17.966771 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:17.966445    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:05:17.966771 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:17.966495    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm podName:59d5b11b-1cb0-411d-bc9a-b16ff5a304c2 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:25.966482053 +0000 UTC m=+285.217019580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-57ksj" (UID: "59d5b11b-1cb0-411d-bc9a-b16ff5a304c2") : configmap "kube-rbac-proxy" not found
Sep 16 12:05:17.966771 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:17.966507    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls podName:59d5b11b-1cb0-411d-bc9a-b16ff5a304c2 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:25.966501954 +0000 UTC m=+285.217039481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-57ksj" (UID: "59d5b11b-1cb0-411d-bc9a-b16ff5a304c2") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:05:21.719962 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:21.719912    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-59d44784f4-6mqnw" podUID="751c95c1-cd17-48f4-9d07-8c7c6c312435"
Sep 16 12:05:22.054518 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:22.054434    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:05:22.101210 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:22.101173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:22.101366 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:22.101237    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:22.101366 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:22.101350    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle podName:82d9505e-b799-4990-9927-e308c84e5292 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:38.101333193 +0000 UTC m=+297.351870719 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle") pod "router-default-69bbc96588-pkg66" (UID: "82d9505e-b799-4990-9927-e308c84e5292") : configmap references non-existent config key: service-ca.crt
Sep 16 12:05:22.101366 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:22.101350    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:05:22.101473 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:22.101387    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs podName:82d9505e-b799-4990-9927-e308c84e5292 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:38.101380035 +0000 UTC m=+297.351917599 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs") pod "router-default-69bbc96588-pkg66" (UID: "82d9505e-b799-4990-9927-e308c84e5292") : secret "router-metrics-certs-default" not found
Sep 16 12:05:25.128208 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:25.128171    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:05:25.128613 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:25.128251    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:05:25.130687 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:25.130660    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aab5b1d1-1603-47a0-a5cc-0d136c7f678a-metrics-tls\") pod \"dns-default-wf5rh\" (UID: \"aab5b1d1-1603-47a0-a5cc-0d136c7f678a\") " pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:05:25.130817 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:25.130709    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls\") pod \"image-registry-59d44784f4-6mqnw\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") " pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:05:25.310265 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:25.310229    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rfzsk\""
Sep 16 12:05:25.317640 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:25.317615    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:05:25.358017 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:25.357989    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pr5zs\""
Sep 16 12:05:25.365932 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:25.365368    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:05:25.587559 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:25.587488    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wf5rh"]
Sep 16 12:05:25.590098 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:05:25.590071    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab5b1d1_1603_47a0_a5cc_0d136c7f678a.slice/crio-ab29feb91c6eff4ca1bbf92d780c61cbf26fe6c26baec00f8d4ba391fd576d62 WatchSource:0}: Error finding container ab29feb91c6eff4ca1bbf92d780c61cbf26fe6c26baec00f8d4ba391fd576d62: Status 404 returned error can't find the container with id ab29feb91c6eff4ca1bbf92d780c61cbf26fe6c26baec00f8d4ba391fd576d62
Sep 16 12:05:25.646291 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:25.646237    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59d44784f4-6mqnw"]
Sep 16 12:05:25.648647 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:05:25.648621    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751c95c1_cd17_48f4_9d07_8c7c6c312435.slice/crio-e9e62557b6b78e3064aad15ab8c2fe2667c35b0f5b2ac6b17485dee17cc635bd WatchSource:0}: Error finding container e9e62557b6b78e3064aad15ab8c2fe2667c35b0f5b2ac6b17485dee17cc635bd: Status 404 returned error can't find the container with id e9e62557b6b78e3064aad15ab8c2fe2667c35b0f5b2ac6b17485dee17cc635bd
Sep 16 12:05:26.036053 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:26.035960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:26.036053 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:26.036028    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:26.036283 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:26.036120    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:05:26.036283 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:26.036190    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls podName:59d5b11b-1cb0-411d-bc9a-b16ff5a304c2 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:42.03617467 +0000 UTC m=+301.286712199 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-57ksj" (UID: "59d5b11b-1cb0-411d-bc9a-b16ff5a304c2") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:05:26.036685 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:26.036661    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:26.067600 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:26.067569    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59d44784f4-6mqnw" event={"ID":"751c95c1-cd17-48f4-9d07-8c7c6c312435","Type":"ContainerStarted","Data":"37d5d115f5531b6929b514906a15174b8b20436fb84c99cd3da95673ddf618ff"}
Sep 16 12:05:26.067806 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:26.067642    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59d44784f4-6mqnw" event={"ID":"751c95c1-cd17-48f4-9d07-8c7c6c312435","Type":"ContainerStarted","Data":"e9e62557b6b78e3064aad15ab8c2fe2667c35b0f5b2ac6b17485dee17cc635bd"}
Sep 16 12:05:26.067806 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:26.067668    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:05:26.068847 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:26.068822    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wf5rh" event={"ID":"aab5b1d1-1603-47a0-a5cc-0d136c7f678a","Type":"ContainerStarted","Data":"ab29feb91c6eff4ca1bbf92d780c61cbf26fe6c26baec00f8d4ba391fd576d62"}
Sep 16 12:05:26.092068 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:26.092024    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59d44784f4-6mqnw" podStartSLOduration=292.092009764 podStartE2EDuration="4m52.092009764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:05:26.091489661 +0000 UTC m=+285.342027210" watchObservedRunningTime="2026-09-16 12:05:26.092009764 +0000 UTC m=+285.342547313"
Sep 16 12:05:28.076900 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:28.076862    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wf5rh" event={"ID":"aab5b1d1-1603-47a0-a5cc-0d136c7f678a","Type":"ContainerStarted","Data":"90537d8785932f9d90714f2d9eaa1ecb08547ff504ecb2ebcfd4d24cb854966a"}
Sep 16 12:05:28.077331 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:28.076918    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wf5rh" event={"ID":"aab5b1d1-1603-47a0-a5cc-0d136c7f678a","Type":"ContainerStarted","Data":"0fa4e740206ba8f0c084fad5a1f5a9e126f8a55b7a80259fa1c3032cabfea570"}
Sep 16 12:05:28.077331 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:28.076986    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:05:28.093666 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:28.093615    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wf5rh" podStartSLOduration=251.039593903 podStartE2EDuration="4m13.093594493s" totalImagesPullingTime="2.05400059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:05:28.093300032 +0000 UTC m=+287.343837581" watchObservedRunningTime="2026-09-16 12:05:28.093594493 +0000 UTC m=+287.344132043"
Sep 16 12:05:36.207204 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:05:36.207137    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-bflx7" podUID="9a90e5c3-5fcb-4016-8682-3803c3121620"
Sep 16 12:05:38.082375 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:38.082349    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wf5rh"
Sep 16 12:05:38.134793 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:38.134759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:38.134973 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:38.134834    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:38.135399 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:38.135365    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d9505e-b799-4990-9927-e308c84e5292-service-ca-bundle\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:38.137454 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:38.137429    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d9505e-b799-4990-9927-e308c84e5292-metrics-certs\") pod \"router-default-69bbc96588-pkg66\" (UID: \"82d9505e-b799-4990-9927-e308c84e5292\") " pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:38.150436 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:38.150413    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:38.419558 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:38.419399    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69bbc96588-pkg66"]
Sep 16 12:05:38.422115 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:05:38.422086    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d9505e_b799_4990_9927_e308c84e5292.slice/crio-595a5c86aaa8cad70348efbbb09e3158762980d4dba82e6b5b641f3ee9b36010 WatchSource:0}: Error finding container 595a5c86aaa8cad70348efbbb09e3158762980d4dba82e6b5b641f3ee9b36010: Status 404 returned error can't find the container with id 595a5c86aaa8cad70348efbbb09e3158762980d4dba82e6b5b641f3ee9b36010
Sep 16 12:05:39.113758 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:39.113694    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69bbc96588-pkg66" event={"ID":"82d9505e-b799-4990-9927-e308c84e5292","Type":"ContainerStarted","Data":"db2d38bf9286fbd8d7514196f0d0e943e9b0600d6fb99d614434241fe96af6a5"}
Sep 16 12:05:39.113758 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:39.113764    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69bbc96588-pkg66" event={"ID":"82d9505e-b799-4990-9927-e308c84e5292","Type":"ContainerStarted","Data":"595a5c86aaa8cad70348efbbb09e3158762980d4dba82e6b5b641f3ee9b36010"}
Sep 16 12:05:39.134517 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:39.134457    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-69bbc96588-pkg66" podStartSLOduration=33.134440413 podStartE2EDuration="33.134440413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:05:39.132936242 +0000 UTC m=+298.383473803" watchObservedRunningTime="2026-09-16 12:05:39.134440413 +0000 UTC m=+298.384977961"
Sep 16 12:05:39.151612 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:39.151577    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:39.153984 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:39.153962    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:40.116819 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:40.116786    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:40.118036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:40.118014    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-69bbc96588-pkg66"
Sep 16 12:05:41.122608 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:41.122586    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 12:05:42.060521 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:42.060489    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:42.062938 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:42.062918    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59d5b11b-1cb0-411d-bc9a-b16ff5a304c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57ksj\" (UID: \"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2\") " pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:42.311459 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:42.311371    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lnmtw\""
Sep 16 12:05:42.318912 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:42.318887    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-57ksj"
Sep 16 12:05:42.587442 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:42.587412    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-57ksj"]
Sep 16 12:05:42.590198 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:05:42.590169    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d5b11b_1cb0_411d_bc9a_b16ff5a304c2.slice/crio-f2112c1e07601c6743ce580359aa80076addacf86c4943d1531a02f89575a238 WatchSource:0}: Error finding container f2112c1e07601c6743ce580359aa80076addacf86c4943d1531a02f89575a238: Status 404 returned error can't find the container with id f2112c1e07601c6743ce580359aa80076addacf86c4943d1531a02f89575a238
Sep 16 12:05:42.646251 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:42.646230    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:05:43.131784 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:43.131725    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-57ksj" event={"ID":"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2","Type":"ContainerStarted","Data":"c1fe1dc6f04a10c00c59df42a4f2c2a662f89a63c9f6b3f93eb6f830cefe469b"}
Sep 16 12:05:43.131960 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:43.131806    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-57ksj" event={"ID":"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2","Type":"ContainerStarted","Data":"f2112c1e07601c6743ce580359aa80076addacf86c4943d1531a02f89575a238"}
Sep 16 12:05:44.137370 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:44.137330    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-57ksj" event={"ID":"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2","Type":"ContainerStarted","Data":"5824640ae808aa067fe8bdfc2f1de31c29a8ed8bfb873e05b3fedad148c9e406"}
Sep 16 12:05:45.142870 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:45.142779    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-57ksj" event={"ID":"59d5b11b-1cb0-411d-bc9a-b16ff5a304c2","Type":"ContainerStarted","Data":"fa952407c5ef14336eb7220eb01e4af4645ec9a074d07c18aecd91e782d2eb1a"}
Sep 16 12:05:45.160378 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:45.160334    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-57ksj" podStartSLOduration=32.990028129 podStartE2EDuration="35.160317213s" totalImagesPullingTime="2.170289084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:05:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:05:45.15944915 +0000 UTC m=+304.409986710" watchObservedRunningTime="2026-09-16 12:05:45.160317213 +0000 UTC m=+304.410854762"
Sep 16 12:05:45.370626 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:45.370591    2582 patch_prober.go:28] interesting pod/image-registry-59d44784f4-6mqnw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 12:05:45.370820 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:45.370645    2582 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-59d44784f4-6mqnw" podUID="751c95c1-cd17-48f4-9d07-8c7c6c312435" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:05:47.075693 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:47.075660    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:05:51.207156 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:51.207117    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:05:54.882256 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:05:54.882225    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59d44784f4-6mqnw"]
Sep 16 12:06:19.901989 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:19.901948    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59d44784f4-6mqnw" podUID="751c95c1-cd17-48f4-9d07-8c7c6c312435" containerName="registry" containerID="cri-o://37d5d115f5531b6929b514906a15174b8b20436fb84c99cd3da95673ddf618ff" gracePeriod=30
Sep 16 12:06:20.145764 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.145723    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:06:20.235065 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.234976    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/751c95c1-cd17-48f4-9d07-8c7c6c312435-trusted-ca\") pod \"751c95c1-cd17-48f4-9d07-8c7c6c312435\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") "
Sep 16 12:06:20.235065 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.235017    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls\") pod \"751c95c1-cd17-48f4-9d07-8c7c6c312435\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") "
Sep 16 12:06:20.235065 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.235041    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-certificates\") pod \"751c95c1-cd17-48f4-9d07-8c7c6c312435\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") "
Sep 16 12:06:20.235307 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.235074    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/751c95c1-cd17-48f4-9d07-8c7c6c312435-image-registry-private-configuration\") pod \"751c95c1-cd17-48f4-9d07-8c7c6c312435\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") "
Sep 16 12:06:20.235307 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.235139    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-bound-sa-token\") pod \"751c95c1-cd17-48f4-9d07-8c7c6c312435\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") "
Sep 16 12:06:20.235307 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.235180    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jcwg\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-kube-api-access-8jcwg\") pod \"751c95c1-cd17-48f4-9d07-8c7c6c312435\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") "
Sep 16 12:06:20.235307 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.235207    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/751c95c1-cd17-48f4-9d07-8c7c6c312435-installation-pull-secrets\") pod \"751c95c1-cd17-48f4-9d07-8c7c6c312435\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") "
Sep 16 12:06:20.235307 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.235236    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/751c95c1-cd17-48f4-9d07-8c7c6c312435-ca-trust-extracted\") pod \"751c95c1-cd17-48f4-9d07-8c7c6c312435\" (UID: \"751c95c1-cd17-48f4-9d07-8c7c6c312435\") "
Sep 16 12:06:20.235717 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.235625    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751c95c1-cd17-48f4-9d07-8c7c6c312435-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "751c95c1-cd17-48f4-9d07-8c7c6c312435" (UID: "751c95c1-cd17-48f4-9d07-8c7c6c312435"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:06:20.235717 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.235634    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "751c95c1-cd17-48f4-9d07-8c7c6c312435" (UID: "751c95c1-cd17-48f4-9d07-8c7c6c312435"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:06:20.236096 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.235821    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/751c95c1-cd17-48f4-9d07-8c7c6c312435-trusted-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:06:20.236096 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.235844    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-certificates\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:06:20.238036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.237999    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "751c95c1-cd17-48f4-9d07-8c7c6c312435" (UID: "751c95c1-cd17-48f4-9d07-8c7c6c312435"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:06:20.238165 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.238121    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "751c95c1-cd17-48f4-9d07-8c7c6c312435" (UID: "751c95c1-cd17-48f4-9d07-8c7c6c312435"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:06:20.238165 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.238155    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/751c95c1-cd17-48f4-9d07-8c7c6c312435-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "751c95c1-cd17-48f4-9d07-8c7c6c312435" (UID: "751c95c1-cd17-48f4-9d07-8c7c6c312435"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:06:20.238282 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.238236    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/751c95c1-cd17-48f4-9d07-8c7c6c312435-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "751c95c1-cd17-48f4-9d07-8c7c6c312435" (UID: "751c95c1-cd17-48f4-9d07-8c7c6c312435"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:06:20.238282 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.238242    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-kube-api-access-8jcwg" (OuterVolumeSpecName: "kube-api-access-8jcwg") pod "751c95c1-cd17-48f4-9d07-8c7c6c312435" (UID: "751c95c1-cd17-48f4-9d07-8c7c6c312435"). InnerVolumeSpecName "kube-api-access-8jcwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:06:20.243999 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.243976    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751c95c1-cd17-48f4-9d07-8c7c6c312435-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "751c95c1-cd17-48f4-9d07-8c7c6c312435" (UID: "751c95c1-cd17-48f4-9d07-8c7c6c312435"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:06:20.254957 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.254931    2582 generic.go:356] "Generic (PLEG): container finished" podID="751c95c1-cd17-48f4-9d07-8c7c6c312435" containerID="37d5d115f5531b6929b514906a15174b8b20436fb84c99cd3da95673ddf618ff" exitCode=0
Sep 16 12:06:20.255061 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.254982    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59d44784f4-6mqnw"
Sep 16 12:06:20.255061 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.254989    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59d44784f4-6mqnw" event={"ID":"751c95c1-cd17-48f4-9d07-8c7c6c312435","Type":"ContainerDied","Data":"37d5d115f5531b6929b514906a15174b8b20436fb84c99cd3da95673ddf618ff"}
Sep 16 12:06:20.255061 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.255032    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59d44784f4-6mqnw" event={"ID":"751c95c1-cd17-48f4-9d07-8c7c6c312435","Type":"ContainerDied","Data":"e9e62557b6b78e3064aad15ab8c2fe2667c35b0f5b2ac6b17485dee17cc635bd"}
Sep 16 12:06:20.255208 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.255063    2582 scope.go:117] "RemoveContainer" containerID="37d5d115f5531b6929b514906a15174b8b20436fb84c99cd3da95673ddf618ff"
Sep 16 12:06:20.263900 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.263882    2582 scope.go:117] "RemoveContainer" containerID="37d5d115f5531b6929b514906a15174b8b20436fb84c99cd3da95673ddf618ff"
Sep 16 12:06:20.264201 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:06:20.264182    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d5d115f5531b6929b514906a15174b8b20436fb84c99cd3da95673ddf618ff\": container with ID starting with 37d5d115f5531b6929b514906a15174b8b20436fb84c99cd3da95673ddf618ff not found: ID does not exist" containerID="37d5d115f5531b6929b514906a15174b8b20436fb84c99cd3da95673ddf618ff"
Sep 16 12:06:20.264255 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.264210    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d5d115f5531b6929b514906a15174b8b20436fb84c99cd3da95673ddf618ff"} err="failed to get container status \"37d5d115f5531b6929b514906a15174b8b20436fb84c99cd3da95673ddf618ff\": rpc error: code = NotFound desc = could not find container \"37d5d115f5531b6929b514906a15174b8b20436fb84c99cd3da95673ddf618ff\": container with ID starting with 37d5d115f5531b6929b514906a15174b8b20436fb84c99cd3da95673ddf618ff not found: ID does not exist"
Sep 16 12:06:20.279151 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.279122    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59d44784f4-6mqnw"]
Sep 16 12:06:20.283177 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.283157    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59d44784f4-6mqnw"]
Sep 16 12:06:20.336842 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.336812    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-bound-sa-token\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:06:20.336842 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.336839    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jcwg\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-kube-api-access-8jcwg\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:06:20.336842 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.336851    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/751c95c1-cd17-48f4-9d07-8c7c6c312435-installation-pull-secrets\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:06:20.337171 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.336861    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/751c95c1-cd17-48f4-9d07-8c7c6c312435-ca-trust-extracted\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:06:20.337171 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.336870    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/751c95c1-cd17-48f4-9d07-8c7c6c312435-registry-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:06:20.337171 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:20.336880    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/751c95c1-cd17-48f4-9d07-8c7c6c312435-image-registry-private-configuration\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:06:21.210511 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:21.210475    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="751c95c1-cd17-48f4-9d07-8c7c6c312435" path="/var/lib/kubelet/pods/751c95c1-cd17-48f4-9d07-8c7c6c312435/volumes"
Sep 16 12:06:40.319047 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:40.319011    2582 generic.go:356] "Generic (PLEG): container finished" podID="747ed2fa-e045-4333-a75c-20d76a36ecf6" containerID="9541acb19e45c2aa408bd3de854a912018f6a2d0b1509efbadc41faebc6af047" exitCode=0
Sep 16 12:06:40.319437 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:40.319086    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9hpds" event={"ID":"747ed2fa-e045-4333-a75c-20d76a36ecf6","Type":"ContainerDied","Data":"9541acb19e45c2aa408bd3de854a912018f6a2d0b1509efbadc41faebc6af047"}
Sep 16 12:06:40.319437 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:40.319422    2582 scope.go:117] "RemoveContainer" containerID="9541acb19e45c2aa408bd3de854a912018f6a2d0b1509efbadc41faebc6af047"
Sep 16 12:06:41.324544 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:41.324508    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9hpds" event={"ID":"747ed2fa-e045-4333-a75c-20d76a36ecf6","Type":"ContainerStarted","Data":"49f1eda5b0377d017d20b6c7a575372ae19e2d147083c920c8ce8beafd647de9"}
Sep 16 12:06:50.903138 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:50.903098    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58d8b45c68-spjc8"]
Sep 16 12:06:50.903769 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:50.903419    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="751c95c1-cd17-48f4-9d07-8c7c6c312435" containerName="registry"
Sep 16 12:06:50.903769 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:50.903436    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="751c95c1-cd17-48f4-9d07-8c7c6c312435" containerName="registry"
Sep 16 12:06:50.903769 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:50.903510    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="751c95c1-cd17-48f4-9d07-8c7c6c312435" containerName="registry"
Sep 16 12:06:50.907790 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:50.907761    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:50.910899 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:50.910876    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 12:06:50.911043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:50.911027    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 12:06:50.911978 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:50.911955    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 12:06:50.911978 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:50.911956    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ftjz5\""
Sep 16 12:06:50.912168 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:50.911958    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 12:06:50.912328 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:50.912312    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 12:06:50.912412 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:50.912393    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 12:06:50.912481 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:50.912426    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 12:06:50.939615 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:50.939584    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58d8b45c68-spjc8"]
Sep 16 12:06:51.066388 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.066351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-service-ca\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.066388 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.066391    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-oauth-serving-cert\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.066627 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.066420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz6d9\" (UniqueName: \"kubernetes.io/projected/8e39d3b9-430b-438b-b173-c109b2c01c3f-kube-api-access-sz6d9\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.066627 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.066456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-oauth-config\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.066627 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.066484    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-config\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.066627 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.066514    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-serving-cert\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.167620 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.167507    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz6d9\" (UniqueName: \"kubernetes.io/projected/8e39d3b9-430b-438b-b173-c109b2c01c3f-kube-api-access-sz6d9\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.167620 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.167561    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-oauth-config\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.167899 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.167648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-config\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.167899 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.167684    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-serving-cert\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.167899 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.167755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-service-ca\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.167899 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.167798    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-oauth-serving-cert\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.168440 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.168412    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-config\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.168440 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.168417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-service-ca\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.168719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.168698    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-oauth-serving-cert\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.170198 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.170178    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-oauth-config\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.170384 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.170364    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-serving-cert\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.176180 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.176160    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz6d9\" (UniqueName: \"kubernetes.io/projected/8e39d3b9-430b-438b-b173-c109b2c01c3f-kube-api-access-sz6d9\") pod \"console-58d8b45c68-spjc8\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") " pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.216626 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.216593    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:06:51.494505 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:51.494420    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58d8b45c68-spjc8"]
Sep 16 12:06:51.499821 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:06:51.499785    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e39d3b9_430b_438b_b173_c109b2c01c3f.slice/crio-594f340586a791bb5949a0e5d93d03a1825ef1eb18004b5a9916e257b3fc7857 WatchSource:0}: Error finding container 594f340586a791bb5949a0e5d93d03a1825ef1eb18004b5a9916e257b3fc7857: Status 404 returned error can't find the container with id 594f340586a791bb5949a0e5d93d03a1825ef1eb18004b5a9916e257b3fc7857
Sep 16 12:06:52.370114 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:52.370076    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d8b45c68-spjc8" event={"ID":"8e39d3b9-430b-438b-b173-c109b2c01c3f","Type":"ContainerStarted","Data":"594f340586a791bb5949a0e5d93d03a1825ef1eb18004b5a9916e257b3fc7857"}
Sep 16 12:06:55.198071 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:55.198031    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs\") pod \"network-metrics-daemon-bflx7\" (UID: \"9a90e5c3-5fcb-4016-8682-3803c3121620\") " pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:06:55.200475 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:55.200456    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a90e5c3-5fcb-4016-8682-3803c3121620-metrics-certs\") pod \"network-metrics-daemon-bflx7\" (UID: \"9a90e5c3-5fcb-4016-8682-3803c3121620\") " pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:06:55.382827 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:55.382795    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d8b45c68-spjc8" event={"ID":"8e39d3b9-430b-438b-b173-c109b2c01c3f","Type":"ContainerStarted","Data":"e231fcf6e505959f3077f6ea5dca932172f8cbcfe70cb52ce02e97db693fb33a"}
Sep 16 12:06:55.410910 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:55.410875    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hp2p6\""
Sep 16 12:06:55.419333 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:55.419295    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bflx7"
Sep 16 12:06:55.687905 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:55.687847    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58d8b45c68-spjc8" podStartSLOduration=2.463211679 podStartE2EDuration="5.687824769s" totalImagesPullingTime="3.22461309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:06:55.401013597 +0000 UTC m=+374.651551145" watchObservedRunningTime="2026-09-16 12:06:55.687824769 +0000 UTC m=+374.938362318"
Sep 16 12:06:55.688377 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:55.688356    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bflx7"]
Sep 16 12:06:55.690716 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:06:55.690691    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a90e5c3_5fcb_4016_8682_3803c3121620.slice/crio-df4e4d190671eb299c0063b0f1313a34ba7a77334ad0bdf13e8d40753fbf5b48 WatchSource:0}: Error finding container df4e4d190671eb299c0063b0f1313a34ba7a77334ad0bdf13e8d40753fbf5b48: Status 404 returned error can't find the container with id df4e4d190671eb299c0063b0f1313a34ba7a77334ad0bdf13e8d40753fbf5b48
Sep 16 12:06:56.388379 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:56.388339    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bflx7" event={"ID":"9a90e5c3-5fcb-4016-8682-3803c3121620","Type":"ContainerStarted","Data":"df4e4d190671eb299c0063b0f1313a34ba7a77334ad0bdf13e8d40753fbf5b48"}
Sep 16 12:06:57.393963 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:57.393919    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bflx7" event={"ID":"9a90e5c3-5fcb-4016-8682-3803c3121620","Type":"ContainerStarted","Data":"e8eae6cc21a9cdba3a0963544997a09ac11215ed6504736a8e387ce96e3f6463"}
Sep 16 12:06:57.394501 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:57.394000    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bflx7" event={"ID":"9a90e5c3-5fcb-4016-8682-3803c3121620","Type":"ContainerStarted","Data":"ec9e486579d04ac27bcd662ed40c9239d061f367349de077580ea693bf887b02"}
Sep 16 12:06:57.410682 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:06:57.410621    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bflx7" podStartSLOduration=375.381521075 podStartE2EDuration="6m16.410604285s" totalImagesPullingTime="1.02908321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:06:57.410160635 +0000 UTC m=+376.660698203" watchObservedRunningTime="2026-09-16 12:06:57.410604285 +0000 UTC m=+376.661141833"
Sep 16 12:07:00.590419 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.590382    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d7d8bc996-fzhm5"]
Sep 16 12:07:00.593523 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.593505    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.603955 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.603933    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 12:07:00.606550 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.606526    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d7d8bc996-fzhm5"]
Sep 16 12:07:00.736315 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.736273    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-trusted-ca-bundle\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.736315 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.736320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-oauth-serving-cert\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.736556 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.736368    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-serving-cert\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.736556 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.736424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-oauth-config\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.736556 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.736468    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgp7m\" (UniqueName: \"kubernetes.io/projected/7b128fab-2b85-40ef-9e3b-543cb654d8dc-kube-api-access-fgp7m\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.736556 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.736545    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-config\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.736717 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.736570    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-service-ca\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.837636 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.837594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgp7m\" (UniqueName: \"kubernetes.io/projected/7b128fab-2b85-40ef-9e3b-543cb654d8dc-kube-api-access-fgp7m\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.837879 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.837657    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-config\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.837879 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.837683    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-service-ca\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.837879 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.837724    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-trusted-ca-bundle\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.837879 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.837813    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-oauth-serving-cert\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.837879 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.837872    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-serving-cert\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.838137 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.837908    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-oauth-config\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.838570 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.838543    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-config\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.838682 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.838543    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-service-ca\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.838682 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.838639    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-oauth-serving-cert\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.838987 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.838964    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-trusted-ca-bundle\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.840463 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.840408    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-serving-cert\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.840602 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.840582    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-oauth-config\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.847472 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.847446    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgp7m\" (UniqueName: \"kubernetes.io/projected/7b128fab-2b85-40ef-9e3b-543cb654d8dc-kube-api-access-fgp7m\") pod \"console-d7d8bc996-fzhm5\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") " pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:00.902518 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:00.902478    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:01.174455 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:01.174375    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d7d8bc996-fzhm5"]
Sep 16 12:07:01.176857 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:07:01.176831    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b128fab_2b85_40ef_9e3b_543cb654d8dc.slice/crio-3aaa149783339b503533549f49b296d3daf39046e6e830213533e6a72e118c00 WatchSource:0}: Error finding container 3aaa149783339b503533549f49b296d3daf39046e6e830213533e6a72e118c00: Status 404 returned error can't find the container with id 3aaa149783339b503533549f49b296d3daf39046e6e830213533e6a72e118c00
Sep 16 12:07:01.217332 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:01.217307    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:07:01.217468 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:01.217368    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:07:01.222751 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:01.222711    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:07:01.408586 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:01.408545    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7d8bc996-fzhm5" event={"ID":"7b128fab-2b85-40ef-9e3b-543cb654d8dc","Type":"ContainerStarted","Data":"95ae16d21e6b8d2f989529bcb7406644d0b0ee7f1e128e9b91f1510ac8fe89e2"}
Sep 16 12:07:01.408794 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:01.408597    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7d8bc996-fzhm5" event={"ID":"7b128fab-2b85-40ef-9e3b-543cb654d8dc","Type":"ContainerStarted","Data":"3aaa149783339b503533549f49b296d3daf39046e6e830213533e6a72e118c00"}
Sep 16 12:07:01.412562 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:01.412535    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:07:01.427180 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:01.427089    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d7d8bc996-fzhm5" podStartSLOduration=1.427073441 podStartE2EDuration="1.427073441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:01.426354985 +0000 UTC m=+380.676892534" watchObservedRunningTime="2026-09-16 12:07:01.427073441 +0000 UTC m=+380.677611011"
Sep 16 12:07:10.903471 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:10.903427    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:10.904033 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:10.903503    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:10.908351 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:10.908327    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:11.447965 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:11.447937    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:07:11.494337 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:11.494299    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58d8b45c68-spjc8"]
Sep 16 12:07:25.310900 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:25.310859    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nms9k"]
Sep 16 12:07:25.314290 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:25.314267    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nms9k"
Sep 16 12:07:25.317249 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:25.317223    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 12:07:25.317397 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:25.317334    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2xdl2\""
Sep 16 12:07:25.338216 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:25.338178    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nms9k"]
Sep 16 12:07:25.421438 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:25.421405    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7dfce9d4-6d98-4673-81eb-76c55fef3ccf-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-nms9k\" (UID: \"7dfce9d4-6d98-4673-81eb-76c55fef3ccf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nms9k"
Sep 16 12:07:25.521887 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:25.521847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7dfce9d4-6d98-4673-81eb-76c55fef3ccf-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-nms9k\" (UID: \"7dfce9d4-6d98-4673-81eb-76c55fef3ccf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nms9k"
Sep 16 12:07:25.522064 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:07:25.521993    2582 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 12:07:25.522064 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:07:25.522060    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dfce9d4-6d98-4673-81eb-76c55fef3ccf-tls-certificates podName:7dfce9d4-6d98-4673-81eb-76c55fef3ccf nodeName:}" failed. No retries permitted until 2026-09-16 12:07:26.022043142 +0000 UTC m=+405.272580669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/7dfce9d4-6d98-4673-81eb-76c55fef3ccf-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-nms9k" (UID: "7dfce9d4-6d98-4673-81eb-76c55fef3ccf") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 12:07:26.026595 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:26.026553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7dfce9d4-6d98-4673-81eb-76c55fef3ccf-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-nms9k\" (UID: \"7dfce9d4-6d98-4673-81eb-76c55fef3ccf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nms9k"
Sep 16 12:07:26.029272 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:26.029248    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7dfce9d4-6d98-4673-81eb-76c55fef3ccf-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-nms9k\" (UID: \"7dfce9d4-6d98-4673-81eb-76c55fef3ccf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nms9k"
Sep 16 12:07:26.223563 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:26.223509    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nms9k"
Sep 16 12:07:26.493600 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:26.493576    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nms9k"]
Sep 16 12:07:26.496075 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:07:26.496050    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfce9d4_6d98_4673_81eb_76c55fef3ccf.slice/crio-23035b17d94fd3dd690900137d19ccc2d701a0b1c5c0cba304a625168794f400 WatchSource:0}: Error finding container 23035b17d94fd3dd690900137d19ccc2d701a0b1c5c0cba304a625168794f400: Status 404 returned error can't find the container with id 23035b17d94fd3dd690900137d19ccc2d701a0b1c5c0cba304a625168794f400
Sep 16 12:07:27.499349 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:27.499308    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nms9k" event={"ID":"7dfce9d4-6d98-4673-81eb-76c55fef3ccf","Type":"ContainerStarted","Data":"23035b17d94fd3dd690900137d19ccc2d701a0b1c5c0cba304a625168794f400"}
Sep 16 12:07:28.504945 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:28.504913    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nms9k" event={"ID":"7dfce9d4-6d98-4673-81eb-76c55fef3ccf","Type":"ContainerStarted","Data":"00d666af4b6468fa88731e9159ed4bdd6c20cdc286866b55a7a9d0f343b0a7ec"}
Sep 16 12:07:28.505397 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:28.505133    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nms9k"
Sep 16 12:07:28.509954 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:28.509931    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nms9k"
Sep 16 12:07:28.521823 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:28.521775    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nms9k" podStartSLOduration=2.251737033 podStartE2EDuration="3.521759467s" totalImagesPullingTime="1.270022434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:28.520165115 +0000 UTC m=+407.770702688" watchObservedRunningTime="2026-09-16 12:07:28.521759467 +0000 UTC m=+407.772297015"
Sep 16 12:07:33.760696 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.760661    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8"]
Sep 16 12:07:33.763866 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.763847    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8"
Sep 16 12:07:33.766944 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.766925    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 12:07:33.767064 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.766965    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-l88rr\""
Sep 16 12:07:33.767365 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.767350    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 12:07:33.768387 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.768369    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 12:07:33.768491 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.768371    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 12:07:33.768491 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.768431    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 12:07:33.774860 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.774841    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/485a8319-25f9-4f8c-a321-aae274ceaa8c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-zt8v8\" (UID: \"485a8319-25f9-4f8c-a321-aae274ceaa8c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8"
Sep 16 12:07:33.774948 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.774877    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz7z6\" (UniqueName: \"kubernetes.io/projected/485a8319-25f9-4f8c-a321-aae274ceaa8c-kube-api-access-mz7z6\") pod \"openshift-state-metrics-67dcf8c979-zt8v8\" (UID: \"485a8319-25f9-4f8c-a321-aae274ceaa8c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8"
Sep 16 12:07:33.774948 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.774902    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/485a8319-25f9-4f8c-a321-aae274ceaa8c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-zt8v8\" (UID: \"485a8319-25f9-4f8c-a321-aae274ceaa8c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8"
Sep 16 12:07:33.775017 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.774994    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/485a8319-25f9-4f8c-a321-aae274ceaa8c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-zt8v8\" (UID: \"485a8319-25f9-4f8c-a321-aae274ceaa8c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8"
Sep 16 12:07:33.778854 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.778833    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8"]
Sep 16 12:07:33.783683 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.783661    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w4qth"]
Sep 16 12:07:33.786726 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.786708    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.788709 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.788688    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"]
Sep 16 12:07:33.789400 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.789379    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rk4zl\""
Sep 16 12:07:33.789400 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.789384    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 12:07:33.789565 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.789531    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 12:07:33.789619 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.789534    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 12:07:33.791865 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.791847    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:33.796172 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.796155    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 12:07:33.796516 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.796495    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 12:07:33.796609 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.796517    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 12:07:33.797002 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.796986    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-d8kpw\""
Sep 16 12:07:33.804499 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.804475    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"]
Sep 16 12:07:33.875647 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.875621    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e603c80f-285b-402e-a4a4-dedc28d07e6e-node-exporter-textfile\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.875647 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.875649    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnm5j\" (UniqueName: \"kubernetes.io/projected/e603c80f-285b-402e-a4a4-dedc28d07e6e-kube-api-access-fnm5j\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.875904 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.875672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e603c80f-285b-402e-a4a4-dedc28d07e6e-node-exporter-accelerators-collector-config\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.875904 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.875720    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e603c80f-285b-402e-a4a4-dedc28d07e6e-node-exporter-tls\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.876216 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.875789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/485a8319-25f9-4f8c-a321-aae274ceaa8c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-zt8v8\" (UID: \"485a8319-25f9-4f8c-a321-aae274ceaa8c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8"
Sep 16 12:07:33.876712 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.876683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e603c80f-285b-402e-a4a4-dedc28d07e6e-root\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.876953 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.876933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz7z6\" (UniqueName: \"kubernetes.io/projected/485a8319-25f9-4f8c-a321-aae274ceaa8c-kube-api-access-mz7z6\") pod \"openshift-state-metrics-67dcf8c979-zt8v8\" (UID: \"485a8319-25f9-4f8c-a321-aae274ceaa8c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8"
Sep 16 12:07:33.877096 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.877081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/485a8319-25f9-4f8c-a321-aae274ceaa8c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-zt8v8\" (UID: \"485a8319-25f9-4f8c-a321-aae274ceaa8c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8"
Sep 16 12:07:33.877217 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.877203    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e603c80f-285b-402e-a4a4-dedc28d07e6e-metrics-client-ca\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.877319 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.877306    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e603c80f-285b-402e-a4a4-dedc28d07e6e-node-exporter-wtmp\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.877418 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.877405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/485a8319-25f9-4f8c-a321-aae274ceaa8c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-zt8v8\" (UID: \"485a8319-25f9-4f8c-a321-aae274ceaa8c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8"
Sep 16 12:07:33.877509 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.877496    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e603c80f-285b-402e-a4a4-dedc28d07e6e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.877613 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.877601    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e603c80f-285b-402e-a4a4-dedc28d07e6e-sys\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.878471 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.878447    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/485a8319-25f9-4f8c-a321-aae274ceaa8c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-zt8v8\" (UID: \"485a8319-25f9-4f8c-a321-aae274ceaa8c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8"
Sep 16 12:07:33.882004 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.881976    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/485a8319-25f9-4f8c-a321-aae274ceaa8c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-zt8v8\" (UID: \"485a8319-25f9-4f8c-a321-aae274ceaa8c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8"
Sep 16 12:07:33.883332 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.883310    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/485a8319-25f9-4f8c-a321-aae274ceaa8c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-zt8v8\" (UID: \"485a8319-25f9-4f8c-a321-aae274ceaa8c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8"
Sep 16 12:07:33.886088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.886062    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz7z6\" (UniqueName: \"kubernetes.io/projected/485a8319-25f9-4f8c-a321-aae274ceaa8c-kube-api-access-mz7z6\") pod \"openshift-state-metrics-67dcf8c979-zt8v8\" (UID: \"485a8319-25f9-4f8c-a321-aae274ceaa8c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8"
Sep 16 12:07:33.978461 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.978421    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e603c80f-285b-402e-a4a4-dedc28d07e6e-sys\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.978461 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.978463    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1af8115e-e839-46e5-9297-22d894d741cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:33.978725 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.978488    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e603c80f-285b-402e-a4a4-dedc28d07e6e-node-exporter-textfile\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.978725 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.978504    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnm5j\" (UniqueName: \"kubernetes.io/projected/e603c80f-285b-402e-a4a4-dedc28d07e6e-kube-api-access-fnm5j\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.978725 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.978536    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e603c80f-285b-402e-a4a4-dedc28d07e6e-sys\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.978725 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.978582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1af8115e-e839-46e5-9297-22d894d741cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:33.978725 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.978628    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e603c80f-285b-402e-a4a4-dedc28d07e6e-node-exporter-accelerators-collector-config\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.978725 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.978662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e603c80f-285b-402e-a4a4-dedc28d07e6e-node-exporter-tls\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.978725 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.978691    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e603c80f-285b-402e-a4a4-dedc28d07e6e-root\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.978725 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.978714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1af8115e-e839-46e5-9297-22d894d741cb-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:33.979095 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.978785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1af8115e-e839-46e5-9297-22d894d741cb-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:33.979095 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.978799    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e603c80f-285b-402e-a4a4-dedc28d07e6e-root\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.979095 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.978829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6txwc\" (UniqueName: \"kubernetes.io/projected/1af8115e-e839-46e5-9297-22d894d741cb-kube-api-access-6txwc\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:33.979095 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.978864    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1af8115e-e839-46e5-9297-22d894d741cb-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:33.979095 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.978928    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e603c80f-285b-402e-a4a4-dedc28d07e6e-node-exporter-textfile\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.979095 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.978930    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e603c80f-285b-402e-a4a4-dedc28d07e6e-metrics-client-ca\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.979095 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.978991    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e603c80f-285b-402e-a4a4-dedc28d07e6e-node-exporter-wtmp\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.979095 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.979026    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e603c80f-285b-402e-a4a4-dedc28d07e6e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.979394 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.979122    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e603c80f-285b-402e-a4a4-dedc28d07e6e-node-exporter-wtmp\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.979394 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.979177    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e603c80f-285b-402e-a4a4-dedc28d07e6e-node-exporter-accelerators-collector-config\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.979394 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.979324    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e603c80f-285b-402e-a4a4-dedc28d07e6e-metrics-client-ca\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.981224 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.981207    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e603c80f-285b-402e-a4a4-dedc28d07e6e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.981273 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.981230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e603c80f-285b-402e-a4a4-dedc28d07e6e-node-exporter-tls\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:33.989550 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:33.989525    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnm5j\" (UniqueName: \"kubernetes.io/projected/e603c80f-285b-402e-a4a4-dedc28d07e6e-kube-api-access-fnm5j\") pod \"node-exporter-w4qth\" (UID: \"e603c80f-285b-402e-a4a4-dedc28d07e6e\") " pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:34.073126 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.073096    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8"
Sep 16 12:07:34.081282 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.081252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1af8115e-e839-46e5-9297-22d894d741cb-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:34.081410 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.081333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1af8115e-e839-46e5-9297-22d894d741cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:34.081410 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.081385    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1af8115e-e839-46e5-9297-22d894d741cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:34.081518 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.081432    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1af8115e-e839-46e5-9297-22d894d741cb-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:34.081518 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.081466    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1af8115e-e839-46e5-9297-22d894d741cb-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:34.081518 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.081494    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6txwc\" (UniqueName: \"kubernetes.io/projected/1af8115e-e839-46e5-9297-22d894d741cb-kube-api-access-6txwc\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:34.082069 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.082035    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1af8115e-e839-46e5-9297-22d894d741cb-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:34.082189 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.082047    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1af8115e-e839-46e5-9297-22d894d741cb-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:34.082189 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.082092    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1af8115e-e839-46e5-9297-22d894d741cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:34.083997 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.083975    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1af8115e-e839-46e5-9297-22d894d741cb-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:34.084207 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.084189    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1af8115e-e839-46e5-9297-22d894d741cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:34.092862 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.092833    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6txwc\" (UniqueName: \"kubernetes.io/projected/1af8115e-e839-46e5-9297-22d894d741cb-kube-api-access-6txwc\") pod \"kube-state-metrics-59fbdd77d-fmxr7\" (UID: \"1af8115e-e839-46e5-9297-22d894d741cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:34.098685 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.098611    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w4qth"
Sep 16 12:07:34.102792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.102767    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"
Sep 16 12:07:34.116097 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:07:34.116068    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode603c80f_285b_402e_a4a4_dedc28d07e6e.slice/crio-8883df556ea5feb13bb5a0550330d27f9c06328f44913c3d31f0a70ab56736bb WatchSource:0}: Error finding container 8883df556ea5feb13bb5a0550330d27f9c06328f44913c3d31f0a70ab56736bb: Status 404 returned error can't find the container with id 8883df556ea5feb13bb5a0550330d27f9c06328f44913c3d31f0a70ab56736bb
Sep 16 12:07:34.352593 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.352515    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8"]
Sep 16 12:07:34.355239 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:07:34.355216    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485a8319_25f9_4f8c_a321_aae274ceaa8c.slice/crio-47ebffac7aa2a4329f281238bf8bfba14349451fd5364e3a3a753a690d0131d6 WatchSource:0}: Error finding container 47ebffac7aa2a4329f281238bf8bfba14349451fd5364e3a3a753a690d0131d6: Status 404 returned error can't find the container with id 47ebffac7aa2a4329f281238bf8bfba14349451fd5364e3a3a753a690d0131d6
Sep 16 12:07:34.398825 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.398797    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7"]
Sep 16 12:07:34.402136 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:07:34.402085    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af8115e_e839_46e5_9297_22d894d741cb.slice/crio-4c33e8f994758bdebc38be9dcfe4d6bcb9186d354337f434f317493c6270940c WatchSource:0}: Error finding container 4c33e8f994758bdebc38be9dcfe4d6bcb9186d354337f434f317493c6270940c: Status 404 returned error can't find the container with id 4c33e8f994758bdebc38be9dcfe4d6bcb9186d354337f434f317493c6270940c
Sep 16 12:07:34.526507 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.526463    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w4qth" event={"ID":"e603c80f-285b-402e-a4a4-dedc28d07e6e","Type":"ContainerStarted","Data":"8883df556ea5feb13bb5a0550330d27f9c06328f44913c3d31f0a70ab56736bb"}
Sep 16 12:07:34.528557 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.528529    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8" event={"ID":"485a8319-25f9-4f8c-a321-aae274ceaa8c","Type":"ContainerStarted","Data":"f00af73063836aa95a1b750cf98bd1b3f85c02486a9e1276f20396c833500390"}
Sep 16 12:07:34.528704 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.528589    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8" event={"ID":"485a8319-25f9-4f8c-a321-aae274ceaa8c","Type":"ContainerStarted","Data":"8be05647f8c236d2a0c157a138872feda6b2664feffa9ea66276dc5eccad7bc2"}
Sep 16 12:07:34.528704 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.528620    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8" event={"ID":"485a8319-25f9-4f8c-a321-aae274ceaa8c","Type":"ContainerStarted","Data":"47ebffac7aa2a4329f281238bf8bfba14349451fd5364e3a3a753a690d0131d6"}
Sep 16 12:07:34.529692 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:34.529663    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7" event={"ID":"1af8115e-e839-46e5-9297-22d894d741cb","Type":"ContainerStarted","Data":"4c33e8f994758bdebc38be9dcfe4d6bcb9186d354337f434f317493c6270940c"}
Sep 16 12:07:36.516791 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.516751    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58d8b45c68-spjc8" podUID="8e39d3b9-430b-438b-b173-c109b2c01c3f" containerName="console" containerID="cri-o://e231fcf6e505959f3077f6ea5dca932172f8cbcfe70cb52ce02e97db693fb33a" gracePeriod=15
Sep 16 12:07:36.539496 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.539462    2582 generic.go:356] "Generic (PLEG): container finished" podID="e603c80f-285b-402e-a4a4-dedc28d07e6e" containerID="f8a867f2d23c8e622aed10308e67b30034131c3f003cc8dca79107040cae29ed" exitCode=0
Sep 16 12:07:36.539652 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.539547    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w4qth" event={"ID":"e603c80f-285b-402e-a4a4-dedc28d07e6e","Type":"ContainerDied","Data":"f8a867f2d23c8e622aed10308e67b30034131c3f003cc8dca79107040cae29ed"}
Sep 16 12:07:36.541906 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.541874    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8" event={"ID":"485a8319-25f9-4f8c-a321-aae274ceaa8c","Type":"ContainerStarted","Data":"fe2ab1f4969aff82134a16c5400cde45c7012bdffef67486ddc7e8dc4b31b877"}
Sep 16 12:07:36.544248 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.544208    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7" event={"ID":"1af8115e-e839-46e5-9297-22d894d741cb","Type":"ContainerStarted","Data":"2e7b790e206e43276a97f6fe410ef64950773a597bfe2f2810e9a9d4a4e5fd7c"}
Sep 16 12:07:36.544343 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.544261    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7" event={"ID":"1af8115e-e839-46e5-9297-22d894d741cb","Type":"ContainerStarted","Data":"a36af8f4e7564e273b2656f6270dd47a285a6be4f964dfb773aeeb1b36907434"}
Sep 16 12:07:36.544343 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.544289    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7" event={"ID":"1af8115e-e839-46e5-9297-22d894d741cb","Type":"ContainerStarted","Data":"8ff1d6a8141e6f82a3b37bc3c814b520d0f9e53f0c9c628c7349317b84fe2e61"}
Sep 16 12:07:36.582214 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.582167    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fmxr7" podStartSLOduration=1.826286233 podStartE2EDuration="3.582150356s" totalImagesPullingTime="1.755864123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:36.581254053 +0000 UTC m=+415.831791603" watchObservedRunningTime="2026-09-16 12:07:36.582150356 +0000 UTC m=+415.832687906"
Sep 16 12:07:36.601374 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.601323    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zt8v8" podStartSLOduration=1.910094531 podStartE2EDuration="3.601303258s" totalImagesPullingTime="1.691208727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:36.599844904 +0000 UTC m=+415.850382460" watchObservedRunningTime="2026-09-16 12:07:36.601303258 +0000 UTC m=+415.851840807"
Sep 16 12:07:36.760869 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.760846    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:07:36.802521 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.802483    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-serving-cert\") pod \"8e39d3b9-430b-438b-b173-c109b2c01c3f\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") "
Sep 16 12:07:36.802690 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.802541    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz6d9\" (UniqueName: \"kubernetes.io/projected/8e39d3b9-430b-438b-b173-c109b2c01c3f-kube-api-access-sz6d9\") pod \"8e39d3b9-430b-438b-b173-c109b2c01c3f\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") "
Sep 16 12:07:36.802690 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.802573    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-oauth-serving-cert\") pod \"8e39d3b9-430b-438b-b173-c109b2c01c3f\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") "
Sep 16 12:07:36.802690 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.802604    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-oauth-config\") pod \"8e39d3b9-430b-438b-b173-c109b2c01c3f\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") "
Sep 16 12:07:36.802887 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.802768    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-config\") pod \"8e39d3b9-430b-438b-b173-c109b2c01c3f\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") "
Sep 16 12:07:36.802887 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.802821    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-service-ca\") pod \"8e39d3b9-430b-438b-b173-c109b2c01c3f\" (UID: \"8e39d3b9-430b-438b-b173-c109b2c01c3f\") "
Sep 16 12:07:36.803057 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.803022    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8e39d3b9-430b-438b-b173-c109b2c01c3f" (UID: "8e39d3b9-430b-438b-b173-c109b2c01c3f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:07:36.803173 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.803127    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-config" (OuterVolumeSpecName: "console-config") pod "8e39d3b9-430b-438b-b173-c109b2c01c3f" (UID: "8e39d3b9-430b-438b-b173-c109b2c01c3f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:07:36.803501 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.803307    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-service-ca" (OuterVolumeSpecName: "service-ca") pod "8e39d3b9-430b-438b-b173-c109b2c01c3f" (UID: "8e39d3b9-430b-438b-b173-c109b2c01c3f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:07:36.805051 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.805029    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e39d3b9-430b-438b-b173-c109b2c01c3f-kube-api-access-sz6d9" (OuterVolumeSpecName: "kube-api-access-sz6d9") pod "8e39d3b9-430b-438b-b173-c109b2c01c3f" (UID: "8e39d3b9-430b-438b-b173-c109b2c01c3f"). InnerVolumeSpecName "kube-api-access-sz6d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:07:36.805137 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.805115    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8e39d3b9-430b-438b-b173-c109b2c01c3f" (UID: "8e39d3b9-430b-438b-b173-c109b2c01c3f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:07:36.805176 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.805130    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8e39d3b9-430b-438b-b173-c109b2c01c3f" (UID: "8e39d3b9-430b-438b-b173-c109b2c01c3f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:07:36.904217 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.904178    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-serving-cert\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:07:36.904217 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.904208    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz6d9\" (UniqueName: \"kubernetes.io/projected/8e39d3b9-430b-438b-b173-c109b2c01c3f-kube-api-access-sz6d9\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:07:36.904217 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.904217    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-oauth-serving-cert\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:07:36.904217 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.904225    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-oauth-config\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:07:36.904483 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.904236    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-console-config\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:07:36.904483 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:36.904257    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e39d3b9-430b-438b-b173-c109b2c01c3f-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:07:37.553287 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:37.553207    2582 generic.go:356] "Generic (PLEG): container finished" podID="8e39d3b9-430b-438b-b173-c109b2c01c3f" containerID="e231fcf6e505959f3077f6ea5dca932172f8cbcfe70cb52ce02e97db693fb33a" exitCode=0
Sep 16 12:07:37.553287 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:37.553259    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d8b45c68-spjc8"
Sep 16 12:07:37.553755 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:37.553283    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d8b45c68-spjc8" event={"ID":"8e39d3b9-430b-438b-b173-c109b2c01c3f","Type":"ContainerDied","Data":"e231fcf6e505959f3077f6ea5dca932172f8cbcfe70cb52ce02e97db693fb33a"}
Sep 16 12:07:37.553755 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:37.553357    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d8b45c68-spjc8" event={"ID":"8e39d3b9-430b-438b-b173-c109b2c01c3f","Type":"ContainerDied","Data":"594f340586a791bb5949a0e5d93d03a1825ef1eb18004b5a9916e257b3fc7857"}
Sep 16 12:07:37.553755 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:37.553375    2582 scope.go:117] "RemoveContainer" containerID="e231fcf6e505959f3077f6ea5dca932172f8cbcfe70cb52ce02e97db693fb33a"
Sep 16 12:07:37.556178 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:37.556154    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w4qth" event={"ID":"e603c80f-285b-402e-a4a4-dedc28d07e6e","Type":"ContainerStarted","Data":"7916f29d312aa674768f96cbeb895e8b3ab8b2276b8396fb36c76982df843eb8"}
Sep 16 12:07:37.556286 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:37.556197    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w4qth" event={"ID":"e603c80f-285b-402e-a4a4-dedc28d07e6e","Type":"ContainerStarted","Data":"703fcfa71aa51211fa169ec7b89d325b975928aa7b72d2b91f582754d6b0d8cc"}
Sep 16 12:07:37.562465 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:37.562448    2582 scope.go:117] "RemoveContainer" containerID="e231fcf6e505959f3077f6ea5dca932172f8cbcfe70cb52ce02e97db693fb33a"
Sep 16 12:07:37.562777 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:07:37.562756    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e231fcf6e505959f3077f6ea5dca932172f8cbcfe70cb52ce02e97db693fb33a\": container with ID starting with e231fcf6e505959f3077f6ea5dca932172f8cbcfe70cb52ce02e97db693fb33a not found: ID does not exist" containerID="e231fcf6e505959f3077f6ea5dca932172f8cbcfe70cb52ce02e97db693fb33a"
Sep 16 12:07:37.562862 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:37.562789    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e231fcf6e505959f3077f6ea5dca932172f8cbcfe70cb52ce02e97db693fb33a"} err="failed to get container status \"e231fcf6e505959f3077f6ea5dca932172f8cbcfe70cb52ce02e97db693fb33a\": rpc error: code = NotFound desc = could not find container \"e231fcf6e505959f3077f6ea5dca932172f8cbcfe70cb52ce02e97db693fb33a\": container with ID starting with e231fcf6e505959f3077f6ea5dca932172f8cbcfe70cb52ce02e97db693fb33a not found: ID does not exist"
Sep 16 12:07:37.569926 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:37.569903    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58d8b45c68-spjc8"]
Sep 16 12:07:37.571899 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:37.571881    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58d8b45c68-spjc8"]
Sep 16 12:07:37.588156 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:37.588116    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w4qth" podStartSLOduration=2.454824849 podStartE2EDuration="4.588103643s" totalImagesPullingTime="2.038620284s" totalInitContainerRuntime="94.65851ms" isStatefulPod=true podCreationTimestamp="2026-09-16 12:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:37.587356181 +0000 UTC m=+416.837893756" watchObservedRunningTime="2026-09-16 12:07:37.588103643 +0000 UTC m=+416.838641191"
Sep 16 12:07:38.530373 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.530335    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-smw9t"]
Sep 16 12:07:38.530664 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.530650    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e39d3b9-430b-438b-b173-c109b2c01c3f" containerName="console"
Sep 16 12:07:38.530705 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.530667    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e39d3b9-430b-438b-b173-c109b2c01c3f" containerName="console"
Sep 16 12:07:38.530765 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.530715    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e39d3b9-430b-438b-b173-c109b2c01c3f" containerName="console"
Sep 16 12:07:38.534953 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.534932    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68f7dddd9d-x6dtk"]
Sep 16 12:07:38.535102 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.535082    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-smw9t"
Sep 16 12:07:38.538104 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.538084    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.538254 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.538231    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 12:07:38.538387 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.538251    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6pbpr\""
Sep 16 12:07:38.540073 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.540052    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-smw9t"]
Sep 16 12:07:38.541208 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.541189    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f7dddd9d-x6dtk"]
Sep 16 12:07:38.619813 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.619770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr7rk\" (UniqueName: \"kubernetes.io/projected/7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32-kube-api-access-jr7rk\") pod \"monitoring-plugin-5d85778b9d-smw9t\" (UID: \"7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-smw9t"
Sep 16 12:07:38.620270 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.619843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e8c0452-4eff-4467-8a42-1eea83903542-console-oauth-config\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.620270 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.619875    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-trusted-ca-bundle\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.620270 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.619944    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e8c0452-4eff-4467-8a42-1eea83903542-console-serving-cert\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.620270 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.619998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-console-config\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.620270 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.620034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-smw9t\" (UID: \"7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-smw9t"
Sep 16 12:07:38.620270 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.620097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-service-ca\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.620270 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.620263    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swdjl\" (UniqueName: \"kubernetes.io/projected/4e8c0452-4eff-4467-8a42-1eea83903542-kube-api-access-swdjl\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.620532 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.620296    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-oauth-serving-cert\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.721447 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.721407    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swdjl\" (UniqueName: \"kubernetes.io/projected/4e8c0452-4eff-4467-8a42-1eea83903542-kube-api-access-swdjl\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.721447 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.721446    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-oauth-serving-cert\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.721711 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.721486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7rk\" (UniqueName: \"kubernetes.io/projected/7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32-kube-api-access-jr7rk\") pod \"monitoring-plugin-5d85778b9d-smw9t\" (UID: \"7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-smw9t"
Sep 16 12:07:38.721711 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.721515    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e8c0452-4eff-4467-8a42-1eea83903542-console-oauth-config\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.721711 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.721541    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-trusted-ca-bundle\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.721711 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.721581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e8c0452-4eff-4467-8a42-1eea83903542-console-serving-cert\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.721711 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.721606    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-console-config\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.721711 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.721637    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-smw9t\" (UID: \"7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-smw9t"
Sep 16 12:07:38.721711 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.721672    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-service-ca\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.722073 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:07:38.721796    2582 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 16 12:07:38.722073 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:07:38.721862    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32-monitoring-plugin-cert podName:7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32 nodeName:}" failed. No retries permitted until 2026-09-16 12:07:39.221840615 +0000 UTC m=+418.472378142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-smw9t" (UID: "7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32") : secret "monitoring-plugin-cert" not found
Sep 16 12:07:38.722406 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.722381    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-oauth-serving-cert\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.722526 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.722506    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-console-config\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.722526 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.722520    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-service-ca\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.722832 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.722809    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-trusted-ca-bundle\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.724333 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.724310    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e8c0452-4eff-4467-8a42-1eea83903542-console-oauth-config\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.724472 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.724454    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e8c0452-4eff-4467-8a42-1eea83903542-console-serving-cert\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.738352 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.738330    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swdjl\" (UniqueName: \"kubernetes.io/projected/4e8c0452-4eff-4467-8a42-1eea83903542-kube-api-access-swdjl\") pod \"console-68f7dddd9d-x6dtk\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") " pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:38.738552 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.738532    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr7rk\" (UniqueName: \"kubernetes.io/projected/7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32-kube-api-access-jr7rk\") pod \"monitoring-plugin-5d85778b9d-smw9t\" (UID: \"7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-smw9t"
Sep 16 12:07:38.851758 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:38.851701    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:39.126959 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:39.126882    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f7dddd9d-x6dtk"]
Sep 16 12:07:39.129709 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:07:39.129685    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8c0452_4eff_4467_8a42_1eea83903542.slice/crio-c968e8fe37da417508215ede3dd3f00e9a5e0ec60547a99f43f892cdcb9eb148 WatchSource:0}: Error finding container c968e8fe37da417508215ede3dd3f00e9a5e0ec60547a99f43f892cdcb9eb148: Status 404 returned error can't find the container with id c968e8fe37da417508215ede3dd3f00e9a5e0ec60547a99f43f892cdcb9eb148
Sep 16 12:07:39.211312 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:39.211280    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e39d3b9-430b-438b-b173-c109b2c01c3f" path="/var/lib/kubelet/pods/8e39d3b9-430b-438b-b173-c109b2c01c3f/volumes"
Sep 16 12:07:39.226445 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:39.226408    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-smw9t\" (UID: \"7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-smw9t"
Sep 16 12:07:39.229069 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:39.229041    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-smw9t\" (UID: \"7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-smw9t"
Sep 16 12:07:39.446927 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:39.446847    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-smw9t"
Sep 16 12:07:39.566014 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:39.565982    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f7dddd9d-x6dtk" event={"ID":"4e8c0452-4eff-4467-8a42-1eea83903542","Type":"ContainerStarted","Data":"fd49b3484bbfb7cca5a3177b9f5b308bf93ac58840df07239a3a11e46c1b1b98"}
Sep 16 12:07:39.566186 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:39.566031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f7dddd9d-x6dtk" event={"ID":"4e8c0452-4eff-4467-8a42-1eea83903542","Type":"ContainerStarted","Data":"c968e8fe37da417508215ede3dd3f00e9a5e0ec60547a99f43f892cdcb9eb148"}
Sep 16 12:07:39.589539 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:39.589488    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68f7dddd9d-x6dtk" podStartSLOduration=1.5894737239999999 podStartE2EDuration="1.589473724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:39.583799635 +0000 UTC m=+418.834337175" watchObservedRunningTime="2026-09-16 12:07:39.589473724 +0000 UTC m=+418.840011274"
Sep 16 12:07:39.713689 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:39.713661    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-smw9t"]
Sep 16 12:07:39.716052 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:07:39.716025    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9d891d_9a19_4bd4_b9da_ffbcefdd0a32.slice/crio-8b08bc13f155ad82fbfc4aa698002b088379b630591a7a0d4c23e2f45b3ff3b3 WatchSource:0}: Error finding container 8b08bc13f155ad82fbfc4aa698002b088379b630591a7a0d4c23e2f45b3ff3b3: Status 404 returned error can't find the container with id 8b08bc13f155ad82fbfc4aa698002b088379b630591a7a0d4c23e2f45b3ff3b3
Sep 16 12:07:40.572875 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:40.572833    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-smw9t" event={"ID":"7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32","Type":"ContainerStarted","Data":"8b08bc13f155ad82fbfc4aa698002b088379b630591a7a0d4c23e2f45b3ff3b3"}
Sep 16 12:07:41.577917 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:41.577879    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-smw9t" event={"ID":"7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32","Type":"ContainerStarted","Data":"620198aeaea73d14f333056f61f9d75d3961e0b1dceaa6943143b88c54bd6250"}
Sep 16 12:07:41.578332 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:41.578090    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-smw9t"
Sep 16 12:07:41.583033 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:41.583011    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-smw9t"
Sep 16 12:07:41.596150 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:41.596105    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-smw9t" podStartSLOduration=1.9913881340000001 podStartE2EDuration="3.596092831s" totalImagesPullingTime="1.604704697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:41.594420338 +0000 UTC m=+420.844957887" watchObservedRunningTime="2026-09-16 12:07:41.596092831 +0000 UTC m=+420.846630381"
Sep 16 12:07:48.852156 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:48.852121    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:48.852550 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:48.852187    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:48.857039 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:48.857009    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:49.615069 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:49.615043    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:07:49.671215 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:07:49.671181    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d7d8bc996-fzhm5"]
Sep 16 12:08:14.697074 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:14.697031    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d7d8bc996-fzhm5" podUID="7b128fab-2b85-40ef-9e3b-543cb654d8dc" containerName="console" containerID="cri-o://95ae16d21e6b8d2f989529bcb7406644d0b0ee7f1e128e9b91f1510ac8fe89e2" gracePeriod=15
Sep 16 12:08:14.947958 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:14.947895    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:08:15.118037 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.117999    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-trusted-ca-bundle\") pod \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") "
Sep 16 12:08:15.118037 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.118042    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-oauth-config\") pod \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") "
Sep 16 12:08:15.118277 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.118094    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-service-ca\") pod \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") "
Sep 16 12:08:15.118277 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.118110    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-config\") pod \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") "
Sep 16 12:08:15.118277 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.118134    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgp7m\" (UniqueName: \"kubernetes.io/projected/7b128fab-2b85-40ef-9e3b-543cb654d8dc-kube-api-access-fgp7m\") pod \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") "
Sep 16 12:08:15.118277 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.118152    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-oauth-serving-cert\") pod \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") "
Sep 16 12:08:15.118277 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.118195    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-serving-cert\") pod \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\" (UID: \"7b128fab-2b85-40ef-9e3b-543cb654d8dc\") "
Sep 16 12:08:15.118611 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.118575    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-service-ca" (OuterVolumeSpecName: "service-ca") pod "7b128fab-2b85-40ef-9e3b-543cb654d8dc" (UID: "7b128fab-2b85-40ef-9e3b-543cb654d8dc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:08:15.118611 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.118593    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7b128fab-2b85-40ef-9e3b-543cb654d8dc" (UID: "7b128fab-2b85-40ef-9e3b-543cb654d8dc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:08:15.118844 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.118700    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-config" (OuterVolumeSpecName: "console-config") pod "7b128fab-2b85-40ef-9e3b-543cb654d8dc" (UID: "7b128fab-2b85-40ef-9e3b-543cb654d8dc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:08:15.118844 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.118772    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7b128fab-2b85-40ef-9e3b-543cb654d8dc" (UID: "7b128fab-2b85-40ef-9e3b-543cb654d8dc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:08:15.120669 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.120637    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b128fab-2b85-40ef-9e3b-543cb654d8dc-kube-api-access-fgp7m" (OuterVolumeSpecName: "kube-api-access-fgp7m") pod "7b128fab-2b85-40ef-9e3b-543cb654d8dc" (UID: "7b128fab-2b85-40ef-9e3b-543cb654d8dc"). InnerVolumeSpecName "kube-api-access-fgp7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:08:15.120669 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.120657    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7b128fab-2b85-40ef-9e3b-543cb654d8dc" (UID: "7b128fab-2b85-40ef-9e3b-543cb654d8dc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:08:15.120983 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.120712    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7b128fab-2b85-40ef-9e3b-543cb654d8dc" (UID: "7b128fab-2b85-40ef-9e3b-543cb654d8dc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:08:15.219693 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.219668    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:08:15.219693 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.219692    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-config\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:08:15.219870 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.219701    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgp7m\" (UniqueName: \"kubernetes.io/projected/7b128fab-2b85-40ef-9e3b-543cb654d8dc-kube-api-access-fgp7m\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:08:15.219870 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.219710    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-oauth-serving-cert\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:08:15.219870 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.219720    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-serving-cert\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:08:15.219870 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.219729    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b128fab-2b85-40ef-9e3b-543cb654d8dc-trusted-ca-bundle\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:08:15.219870 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.219756    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b128fab-2b85-40ef-9e3b-543cb654d8dc-console-oauth-config\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:08:15.713824 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.713781    2582 generic.go:356] "Generic (PLEG): container finished" podID="7b128fab-2b85-40ef-9e3b-543cb654d8dc" containerID="95ae16d21e6b8d2f989529bcb7406644d0b0ee7f1e128e9b91f1510ac8fe89e2" exitCode=0
Sep 16 12:08:15.714325 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.713848    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7d8bc996-fzhm5"
Sep 16 12:08:15.714325 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.713871    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7d8bc996-fzhm5" event={"ID":"7b128fab-2b85-40ef-9e3b-543cb654d8dc","Type":"ContainerDied","Data":"95ae16d21e6b8d2f989529bcb7406644d0b0ee7f1e128e9b91f1510ac8fe89e2"}
Sep 16 12:08:15.714325 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.713968    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7d8bc996-fzhm5" event={"ID":"7b128fab-2b85-40ef-9e3b-543cb654d8dc","Type":"ContainerDied","Data":"3aaa149783339b503533549f49b296d3daf39046e6e830213533e6a72e118c00"}
Sep 16 12:08:15.714325 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.714002    2582 scope.go:117] "RemoveContainer" containerID="95ae16d21e6b8d2f989529bcb7406644d0b0ee7f1e128e9b91f1510ac8fe89e2"
Sep 16 12:08:15.722847 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.722830    2582 scope.go:117] "RemoveContainer" containerID="95ae16d21e6b8d2f989529bcb7406644d0b0ee7f1e128e9b91f1510ac8fe89e2"
Sep 16 12:08:15.723145 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:08:15.723127    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ae16d21e6b8d2f989529bcb7406644d0b0ee7f1e128e9b91f1510ac8fe89e2\": container with ID starting with 95ae16d21e6b8d2f989529bcb7406644d0b0ee7f1e128e9b91f1510ac8fe89e2 not found: ID does not exist" containerID="95ae16d21e6b8d2f989529bcb7406644d0b0ee7f1e128e9b91f1510ac8fe89e2"
Sep 16 12:08:15.723199 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.723153    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ae16d21e6b8d2f989529bcb7406644d0b0ee7f1e128e9b91f1510ac8fe89e2"} err="failed to get container status \"95ae16d21e6b8d2f989529bcb7406644d0b0ee7f1e128e9b91f1510ac8fe89e2\": rpc error: code = NotFound desc = could not find container \"95ae16d21e6b8d2f989529bcb7406644d0b0ee7f1e128e9b91f1510ac8fe89e2\": container with ID starting with 95ae16d21e6b8d2f989529bcb7406644d0b0ee7f1e128e9b91f1510ac8fe89e2 not found: ID does not exist"
Sep 16 12:08:15.730423 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.730391    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d7d8bc996-fzhm5"]
Sep 16 12:08:15.734457 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:15.734436    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d7d8bc996-fzhm5"]
Sep 16 12:08:17.211494 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:08:17.211461    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b128fab-2b85-40ef-9e3b-543cb654d8dc" path="/var/lib/kubelet/pods/7b128fab-2b85-40ef-9e3b-543cb654d8dc/volumes"
Sep 16 12:09:01.651781 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.651727    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-549dcb579d-srss8"]
Sep 16 12:09:01.652334 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.652087    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b128fab-2b85-40ef-9e3b-543cb654d8dc" containerName="console"
Sep 16 12:09:01.652334 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.652102    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b128fab-2b85-40ef-9e3b-543cb654d8dc" containerName="console"
Sep 16 12:09:01.652334 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.652199    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b128fab-2b85-40ef-9e3b-543cb654d8dc" containerName="console"
Sep 16 12:09:01.656597 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.656575    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.673138 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.673112    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-549dcb579d-srss8"]
Sep 16 12:09:01.777831 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.777794    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-trusted-ca-bundle\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.778000 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.777840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-service-ca\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.778000 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.777864    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92r48\" (UniqueName: \"kubernetes.io/projected/cc3014a8-7528-42e9-8fcf-2767bb5f2529-kube-api-access-92r48\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.778000 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.777945    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-config\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.778000 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.777983    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-oauth-serving-cert\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.778135 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.778017    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-oauth-config\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.778135 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.778063    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-serving-cert\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.879086 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.879051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-service-ca\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.879086 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.879084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92r48\" (UniqueName: \"kubernetes.io/projected/cc3014a8-7528-42e9-8fcf-2767bb5f2529-kube-api-access-92r48\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.879273 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.879111    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-config\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.879273 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.879139    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-oauth-serving-cert\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.879273 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.879175    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-oauth-config\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.879273 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.879212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-serving-cert\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.879273 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.879254    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-trusted-ca-bundle\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.879942 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.879918    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-oauth-serving-cert\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.880048 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.879958    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-config\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.880048 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.879958    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-service-ca\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.880388 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.880366    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-trusted-ca-bundle\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.881866 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.881845    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-oauth-config\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.882018 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.882001    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-serving-cert\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.886771 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.886725    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92r48\" (UniqueName: \"kubernetes.io/projected/cc3014a8-7528-42e9-8fcf-2767bb5f2529-kube-api-access-92r48\") pod \"console-549dcb579d-srss8\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") " pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:01.970472 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:01.970377    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:02.243375 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:02.243348    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-549dcb579d-srss8"]
Sep 16 12:09:02.246016 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:09:02.245986    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3014a8_7528_42e9_8fcf_2767bb5f2529.slice/crio-dbd56669665587b921250dbd5bb74c10d4ac1fcee759f0792a1d832b50af85af WatchSource:0}: Error finding container dbd56669665587b921250dbd5bb74c10d4ac1fcee759f0792a1d832b50af85af: Status 404 returned error can't find the container with id dbd56669665587b921250dbd5bb74c10d4ac1fcee759f0792a1d832b50af85af
Sep 16 12:09:02.896200 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:02.896168    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549dcb579d-srss8" event={"ID":"cc3014a8-7528-42e9-8fcf-2767bb5f2529","Type":"ContainerStarted","Data":"0a93bfbf89993e3162ada3dec87720feaa3deaeae38beb8f34add78b732c2db7"}
Sep 16 12:09:02.896556 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:02.896213    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549dcb579d-srss8" event={"ID":"cc3014a8-7528-42e9-8fcf-2767bb5f2529","Type":"ContainerStarted","Data":"dbd56669665587b921250dbd5bb74c10d4ac1fcee759f0792a1d832b50af85af"}
Sep 16 12:09:02.914888 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:02.914839    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-549dcb579d-srss8" podStartSLOduration=1.914824282 podStartE2EDuration="1.914824282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:09:02.913983146 +0000 UTC m=+502.164520694" watchObservedRunningTime="2026-09-16 12:09:02.914824282 +0000 UTC m=+502.165361878"
Sep 16 12:09:11.970576 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:11.970543    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:11.971054 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:11.970624    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:11.975458 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:11.975434    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:12.936817 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:12.936789    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:12.984477 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:12.984433    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68f7dddd9d-x6dtk"]
Sep 16 12:09:13.093771 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.093714    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75f77c54b-lr9h6"]
Sep 16 12:09:13.096981 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.096966    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.109603 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.109576    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75f77c54b-lr9h6"]
Sep 16 12:09:13.171665 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.171632    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c8026c6-c14f-42fb-8664-c525009a114f-console-oauth-config\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.171887 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.171675    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wstxt\" (UniqueName: \"kubernetes.io/projected/2c8026c6-c14f-42fb-8664-c525009a114f-kube-api-access-wstxt\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.171887 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.171695    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c8026c6-c14f-42fb-8664-c525009a114f-console-serving-cert\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.171887 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.171801    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-oauth-serving-cert\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.171887 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.171851    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-trusted-ca-bundle\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.171887 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.171876    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-console-config\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.172054 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.171930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-service-ca\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.272953 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.272874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wstxt\" (UniqueName: \"kubernetes.io/projected/2c8026c6-c14f-42fb-8664-c525009a114f-kube-api-access-wstxt\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.272953 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.272907    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c8026c6-c14f-42fb-8664-c525009a114f-console-serving-cert\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.272953 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.272927    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-oauth-serving-cert\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.273185 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.273060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-trusted-ca-bundle\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.273185 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.273107    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-console-config\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.273185 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.273148    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-service-ca\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.273329 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.273199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c8026c6-c14f-42fb-8664-c525009a114f-console-oauth-config\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.273687 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.273659    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-oauth-serving-cert\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.273861 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.273842    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-console-config\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.273916 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.273877    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-trusted-ca-bundle\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.274145 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.274121    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-service-ca\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.275546 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.275525    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c8026c6-c14f-42fb-8664-c525009a114f-console-oauth-config\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.275916 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.275896    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c8026c6-c14f-42fb-8664-c525009a114f-console-serving-cert\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.281567 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.281547    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wstxt\" (UniqueName: \"kubernetes.io/projected/2c8026c6-c14f-42fb-8664-c525009a114f-kube-api-access-wstxt\") pod \"console-75f77c54b-lr9h6\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") " pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.406790 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.406752    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:13.678753 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.678704    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75f77c54b-lr9h6"]
Sep 16 12:09:13.681760 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:09:13.681705    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8026c6_c14f_42fb_8664_c525009a114f.slice/crio-dc8955d9eb0ea499bbaf6727474f26107493d549bcfa6885dffb59cf7b3d14a9 WatchSource:0}: Error finding container dc8955d9eb0ea499bbaf6727474f26107493d549bcfa6885dffb59cf7b3d14a9: Status 404 returned error can't find the container with id dc8955d9eb0ea499bbaf6727474f26107493d549bcfa6885dffb59cf7b3d14a9
Sep 16 12:09:13.937797 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.937678    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f77c54b-lr9h6" event={"ID":"2c8026c6-c14f-42fb-8664-c525009a114f","Type":"ContainerStarted","Data":"12672d869742543eea4c94cea579b08e72d32e9e258cd01a34d512926fb26266"}
Sep 16 12:09:13.937797 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.937727    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f77c54b-lr9h6" event={"ID":"2c8026c6-c14f-42fb-8664-c525009a114f","Type":"ContainerStarted","Data":"dc8955d9eb0ea499bbaf6727474f26107493d549bcfa6885dffb59cf7b3d14a9"}
Sep 16 12:09:13.956453 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:13.956390    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75f77c54b-lr9h6" podStartSLOduration=0.956370352 podStartE2EDuration="956.370352ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:09:13.955613248 +0000 UTC m=+513.206150797" watchObservedRunningTime="2026-09-16 12:09:13.956370352 +0000 UTC m=+513.206907901"
Sep 16 12:09:23.406900 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:23.406867    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:23.407334 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:23.406923    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:23.411562 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:23.411537    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:23.978184 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:23.978154    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:09:24.020642 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:24.020602    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-549dcb579d-srss8"]
Sep 16 12:09:38.009234 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.009129    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68f7dddd9d-x6dtk" podUID="4e8c0452-4eff-4467-8a42-1eea83903542" containerName="console" containerID="cri-o://fd49b3484bbfb7cca5a3177b9f5b308bf93ac58840df07239a3a11e46c1b1b98" gracePeriod=15
Sep 16 12:09:38.251128 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.251100    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:09:38.281986 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.281911    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-service-ca\") pod \"4e8c0452-4eff-4467-8a42-1eea83903542\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") "
Sep 16 12:09:38.281986 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.281955    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-trusted-ca-bundle\") pod \"4e8c0452-4eff-4467-8a42-1eea83903542\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") "
Sep 16 12:09:38.282196 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.282012    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e8c0452-4eff-4467-8a42-1eea83903542-console-serving-cert\") pod \"4e8c0452-4eff-4467-8a42-1eea83903542\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") "
Sep 16 12:09:38.282196 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.282071    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e8c0452-4eff-4467-8a42-1eea83903542-console-oauth-config\") pod \"4e8c0452-4eff-4467-8a42-1eea83903542\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") "
Sep 16 12:09:38.282196 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.282099    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-oauth-serving-cert\") pod \"4e8c0452-4eff-4467-8a42-1eea83903542\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") "
Sep 16 12:09:38.282370 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.282318    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-service-ca" (OuterVolumeSpecName: "service-ca") pod "4e8c0452-4eff-4467-8a42-1eea83903542" (UID: "4e8c0452-4eff-4467-8a42-1eea83903542"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:38.282480 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.282456    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4e8c0452-4eff-4467-8a42-1eea83903542" (UID: "4e8c0452-4eff-4467-8a42-1eea83903542"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:38.282562 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.282516    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4e8c0452-4eff-4467-8a42-1eea83903542" (UID: "4e8c0452-4eff-4467-8a42-1eea83903542"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:38.284350 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.284327    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e8c0452-4eff-4467-8a42-1eea83903542-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4e8c0452-4eff-4467-8a42-1eea83903542" (UID: "4e8c0452-4eff-4467-8a42-1eea83903542"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:38.284478 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.284448    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e8c0452-4eff-4467-8a42-1eea83903542-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4e8c0452-4eff-4467-8a42-1eea83903542" (UID: "4e8c0452-4eff-4467-8a42-1eea83903542"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:38.383464 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.383430    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-console-config\") pod \"4e8c0452-4eff-4467-8a42-1eea83903542\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") "
Sep 16 12:09:38.383464 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.383467    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swdjl\" (UniqueName: \"kubernetes.io/projected/4e8c0452-4eff-4467-8a42-1eea83903542-kube-api-access-swdjl\") pod \"4e8c0452-4eff-4467-8a42-1eea83903542\" (UID: \"4e8c0452-4eff-4467-8a42-1eea83903542\") "
Sep 16 12:09:38.383727 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.383711    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e8c0452-4eff-4467-8a42-1eea83903542-console-oauth-config\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:09:38.383800 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.383759    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-oauth-serving-cert\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:09:38.383800 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.383775    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:09:38.383800 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.383795    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-trusted-ca-bundle\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:09:38.383899 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.383809    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e8c0452-4eff-4467-8a42-1eea83903542-console-serving-cert\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:09:38.383899 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.383861    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-console-config" (OuterVolumeSpecName: "console-config") pod "4e8c0452-4eff-4467-8a42-1eea83903542" (UID: "4e8c0452-4eff-4467-8a42-1eea83903542"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:38.385661 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.385638    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8c0452-4eff-4467-8a42-1eea83903542-kube-api-access-swdjl" (OuterVolumeSpecName: "kube-api-access-swdjl") pod "4e8c0452-4eff-4467-8a42-1eea83903542" (UID: "4e8c0452-4eff-4467-8a42-1eea83903542"). InnerVolumeSpecName "kube-api-access-swdjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:09:38.484756 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.484708    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e8c0452-4eff-4467-8a42-1eea83903542-console-config\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:09:38.484756 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:38.484753    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swdjl\" (UniqueName: \"kubernetes.io/projected/4e8c0452-4eff-4467-8a42-1eea83903542-kube-api-access-swdjl\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:09:39.034423 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:39.034393    2582 generic.go:356] "Generic (PLEG): container finished" podID="4e8c0452-4eff-4467-8a42-1eea83903542" containerID="fd49b3484bbfb7cca5a3177b9f5b308bf93ac58840df07239a3a11e46c1b1b98" exitCode=0
Sep 16 12:09:39.034853 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:39.034443    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f7dddd9d-x6dtk"
Sep 16 12:09:39.034853 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:39.034489    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f7dddd9d-x6dtk" event={"ID":"4e8c0452-4eff-4467-8a42-1eea83903542","Type":"ContainerDied","Data":"fd49b3484bbfb7cca5a3177b9f5b308bf93ac58840df07239a3a11e46c1b1b98"}
Sep 16 12:09:39.034853 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:39.034542    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f7dddd9d-x6dtk" event={"ID":"4e8c0452-4eff-4467-8a42-1eea83903542","Type":"ContainerDied","Data":"c968e8fe37da417508215ede3dd3f00e9a5e0ec60547a99f43f892cdcb9eb148"}
Sep 16 12:09:39.034853 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:39.034562    2582 scope.go:117] "RemoveContainer" containerID="fd49b3484bbfb7cca5a3177b9f5b308bf93ac58840df07239a3a11e46c1b1b98"
Sep 16 12:09:39.043486 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:39.043463    2582 scope.go:117] "RemoveContainer" containerID="fd49b3484bbfb7cca5a3177b9f5b308bf93ac58840df07239a3a11e46c1b1b98"
Sep 16 12:09:39.043785 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:09:39.043766    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd49b3484bbfb7cca5a3177b9f5b308bf93ac58840df07239a3a11e46c1b1b98\": container with ID starting with fd49b3484bbfb7cca5a3177b9f5b308bf93ac58840df07239a3a11e46c1b1b98 not found: ID does not exist" containerID="fd49b3484bbfb7cca5a3177b9f5b308bf93ac58840df07239a3a11e46c1b1b98"
Sep 16 12:09:39.043846 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:39.043802    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd49b3484bbfb7cca5a3177b9f5b308bf93ac58840df07239a3a11e46c1b1b98"} err="failed to get container status \"fd49b3484bbfb7cca5a3177b9f5b308bf93ac58840df07239a3a11e46c1b1b98\": rpc error: code = NotFound desc = could not find container \"fd49b3484bbfb7cca5a3177b9f5b308bf93ac58840df07239a3a11e46c1b1b98\": container with ID starting with fd49b3484bbfb7cca5a3177b9f5b308bf93ac58840df07239a3a11e46c1b1b98 not found: ID does not exist"
Sep 16 12:09:39.055136 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:39.055112    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68f7dddd9d-x6dtk"]
Sep 16 12:09:39.057251 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:39.057228    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68f7dddd9d-x6dtk"]
Sep 16 12:09:39.210957 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:39.210922    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e8c0452-4eff-4467-8a42-1eea83903542" path="/var/lib/kubelet/pods/4e8c0452-4eff-4467-8a42-1eea83903542/volumes"
Sep 16 12:09:49.047205 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.047142    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-549dcb579d-srss8" podUID="cc3014a8-7528-42e9-8fcf-2767bb5f2529" containerName="console" containerID="cri-o://0a93bfbf89993e3162ada3dec87720feaa3deaeae38beb8f34add78b732c2db7" gracePeriod=15
Sep 16 12:09:49.291010 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.290988    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:49.364444 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.364413    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-oauth-config\") pod \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") "
Sep 16 12:09:49.364651 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.364471    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-trusted-ca-bundle\") pod \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") "
Sep 16 12:09:49.364651 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.364493    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-config\") pod \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") "
Sep 16 12:09:49.364651 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.364528    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-oauth-serving-cert\") pod \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") "
Sep 16 12:09:49.364651 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.364553    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-serving-cert\") pod \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") "
Sep 16 12:09:49.364651 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.364579    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-service-ca\") pod \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") "
Sep 16 12:09:49.364651 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.364611    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92r48\" (UniqueName: \"kubernetes.io/projected/cc3014a8-7528-42e9-8fcf-2767bb5f2529-kube-api-access-92r48\") pod \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\" (UID: \"cc3014a8-7528-42e9-8fcf-2767bb5f2529\") "
Sep 16 12:09:49.365003 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.364978    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-config" (OuterVolumeSpecName: "console-config") pod "cc3014a8-7528-42e9-8fcf-2767bb5f2529" (UID: "cc3014a8-7528-42e9-8fcf-2767bb5f2529"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:49.365062 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.365010    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cc3014a8-7528-42e9-8fcf-2767bb5f2529" (UID: "cc3014a8-7528-42e9-8fcf-2767bb5f2529"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:49.365062 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.365025    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cc3014a8-7528-42e9-8fcf-2767bb5f2529" (UID: "cc3014a8-7528-42e9-8fcf-2767bb5f2529"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:49.365257 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.365229    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-service-ca" (OuterVolumeSpecName: "service-ca") pod "cc3014a8-7528-42e9-8fcf-2767bb5f2529" (UID: "cc3014a8-7528-42e9-8fcf-2767bb5f2529"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:49.366701 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.366680    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cc3014a8-7528-42e9-8fcf-2767bb5f2529" (UID: "cc3014a8-7528-42e9-8fcf-2767bb5f2529"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:49.367092 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.367063    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc3014a8-7528-42e9-8fcf-2767bb5f2529-kube-api-access-92r48" (OuterVolumeSpecName: "kube-api-access-92r48") pod "cc3014a8-7528-42e9-8fcf-2767bb5f2529" (UID: "cc3014a8-7528-42e9-8fcf-2767bb5f2529"). InnerVolumeSpecName "kube-api-access-92r48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:09:49.367295 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.367272    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cc3014a8-7528-42e9-8fcf-2767bb5f2529" (UID: "cc3014a8-7528-42e9-8fcf-2767bb5f2529"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:49.465562 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.465519    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-trusted-ca-bundle\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:09:49.465562 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.465550    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-config\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:09:49.465562 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.465564    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-oauth-serving-cert\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:09:49.465833 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.465575    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-serving-cert\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:09:49.465833 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.465589    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3014a8-7528-42e9-8fcf-2767bb5f2529-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:09:49.465833 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.465600    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92r48\" (UniqueName: \"kubernetes.io/projected/cc3014a8-7528-42e9-8fcf-2767bb5f2529-kube-api-access-92r48\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:09:49.465833 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:49.465611    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc3014a8-7528-42e9-8fcf-2767bb5f2529-console-oauth-config\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:09:50.077792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:50.077753    2582 generic.go:356] "Generic (PLEG): container finished" podID="cc3014a8-7528-42e9-8fcf-2767bb5f2529" containerID="0a93bfbf89993e3162ada3dec87720feaa3deaeae38beb8f34add78b732c2db7" exitCode=0
Sep 16 12:09:50.077792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:50.077794    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549dcb579d-srss8"
Sep 16 12:09:50.078274 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:50.077823    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549dcb579d-srss8" event={"ID":"cc3014a8-7528-42e9-8fcf-2767bb5f2529","Type":"ContainerDied","Data":"0a93bfbf89993e3162ada3dec87720feaa3deaeae38beb8f34add78b732c2db7"}
Sep 16 12:09:50.078274 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:50.077874    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549dcb579d-srss8" event={"ID":"cc3014a8-7528-42e9-8fcf-2767bb5f2529","Type":"ContainerDied","Data":"dbd56669665587b921250dbd5bb74c10d4ac1fcee759f0792a1d832b50af85af"}
Sep 16 12:09:50.078274 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:50.077896    2582 scope.go:117] "RemoveContainer" containerID="0a93bfbf89993e3162ada3dec87720feaa3deaeae38beb8f34add78b732c2db7"
Sep 16 12:09:50.086693 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:50.086679    2582 scope.go:117] "RemoveContainer" containerID="0a93bfbf89993e3162ada3dec87720feaa3deaeae38beb8f34add78b732c2db7"
Sep 16 12:09:50.086991 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:09:50.086973    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a93bfbf89993e3162ada3dec87720feaa3deaeae38beb8f34add78b732c2db7\": container with ID starting with 0a93bfbf89993e3162ada3dec87720feaa3deaeae38beb8f34add78b732c2db7 not found: ID does not exist" containerID="0a93bfbf89993e3162ada3dec87720feaa3deaeae38beb8f34add78b732c2db7"
Sep 16 12:09:50.087058 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:50.087004    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a93bfbf89993e3162ada3dec87720feaa3deaeae38beb8f34add78b732c2db7"} err="failed to get container status \"0a93bfbf89993e3162ada3dec87720feaa3deaeae38beb8f34add78b732c2db7\": rpc error: code = NotFound desc = could not find container \"0a93bfbf89993e3162ada3dec87720feaa3deaeae38beb8f34add78b732c2db7\": container with ID starting with 0a93bfbf89993e3162ada3dec87720feaa3deaeae38beb8f34add78b732c2db7 not found: ID does not exist"
Sep 16 12:09:50.097808 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:50.097784    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-549dcb579d-srss8"]
Sep 16 12:09:50.102318 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:50.102296    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-549dcb579d-srss8"]
Sep 16 12:09:51.210723 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:09:51.210685    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc3014a8-7528-42e9-8fcf-2767bb5f2529" path="/var/lib/kubelet/pods/cc3014a8-7528-42e9-8fcf-2767bb5f2529/volumes"
Sep 16 12:10:21.726937 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.726899    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps"]
Sep 16 12:10:21.727373 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.727199    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e8c0452-4eff-4467-8a42-1eea83903542" containerName="console"
Sep 16 12:10:21.727373 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.727210    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e8c0452-4eff-4467-8a42-1eea83903542" containerName="console"
Sep 16 12:10:21.727373 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.727223    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc3014a8-7528-42e9-8fcf-2767bb5f2529" containerName="console"
Sep 16 12:10:21.727373 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.727229    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc3014a8-7528-42e9-8fcf-2767bb5f2529" containerName="console"
Sep 16 12:10:21.727373 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.727274    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc3014a8-7528-42e9-8fcf-2767bb5f2529" containerName="console"
Sep 16 12:10:21.727373 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.727283    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e8c0452-4eff-4467-8a42-1eea83903542" containerName="console"
Sep 16 12:10:21.730194 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.730178    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps"
Sep 16 12:10:21.733027 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.733003    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bp9w9\""
Sep 16 12:10:21.733158 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.733068    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:10:21.734163 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.734147    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:10:21.739203 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.739181    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps"]
Sep 16 12:10:21.823981 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.823948    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/343ec258-cae8-42ca-a473-5dcc9798e1e3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps\" (UID: \"343ec258-cae8-42ca-a473-5dcc9798e1e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps"
Sep 16 12:10:21.824166 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.823996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6tvh\" (UniqueName: \"kubernetes.io/projected/343ec258-cae8-42ca-a473-5dcc9798e1e3-kube-api-access-p6tvh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps\" (UID: \"343ec258-cae8-42ca-a473-5dcc9798e1e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps"
Sep 16 12:10:21.824166 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.824114    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/343ec258-cae8-42ca-a473-5dcc9798e1e3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps\" (UID: \"343ec258-cae8-42ca-a473-5dcc9798e1e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps"
Sep 16 12:10:21.925080 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.925041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/343ec258-cae8-42ca-a473-5dcc9798e1e3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps\" (UID: \"343ec258-cae8-42ca-a473-5dcc9798e1e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps"
Sep 16 12:10:21.925080 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.925088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6tvh\" (UniqueName: \"kubernetes.io/projected/343ec258-cae8-42ca-a473-5dcc9798e1e3-kube-api-access-p6tvh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps\" (UID: \"343ec258-cae8-42ca-a473-5dcc9798e1e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps"
Sep 16 12:10:21.925324 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.925135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/343ec258-cae8-42ca-a473-5dcc9798e1e3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps\" (UID: \"343ec258-cae8-42ca-a473-5dcc9798e1e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps"
Sep 16 12:10:21.925533 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.925510    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/343ec258-cae8-42ca-a473-5dcc9798e1e3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps\" (UID: \"343ec258-cae8-42ca-a473-5dcc9798e1e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps"
Sep 16 12:10:21.925611 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.925523    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/343ec258-cae8-42ca-a473-5dcc9798e1e3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps\" (UID: \"343ec258-cae8-42ca-a473-5dcc9798e1e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps"
Sep 16 12:10:21.933264 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:21.933245    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6tvh\" (UniqueName: \"kubernetes.io/projected/343ec258-cae8-42ca-a473-5dcc9798e1e3-kube-api-access-p6tvh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps\" (UID: \"343ec258-cae8-42ca-a473-5dcc9798e1e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps"
Sep 16 12:10:22.039979 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:22.039903    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps"
Sep 16 12:10:22.310103 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:22.310079    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps"]
Sep 16 12:10:22.312458 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:10:22.312427    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343ec258_cae8_42ca_a473_5dcc9798e1e3.slice/crio-c4c8028f40f0f50711930ec4b26ea06f8e51a5b9e740ecc44bc1de2563f93adb WatchSource:0}: Error finding container c4c8028f40f0f50711930ec4b26ea06f8e51a5b9e740ecc44bc1de2563f93adb: Status 404 returned error can't find the container with id c4c8028f40f0f50711930ec4b26ea06f8e51a5b9e740ecc44bc1de2563f93adb
Sep 16 12:10:23.211151 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:23.211108    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps" event={"ID":"343ec258-cae8-42ca-a473-5dcc9798e1e3","Type":"ContainerStarted","Data":"c4c8028f40f0f50711930ec4b26ea06f8e51a5b9e740ecc44bc1de2563f93adb"}
Sep 16 12:10:29.234605 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:29.234517    2582 generic.go:356] "Generic (PLEG): container finished" podID="343ec258-cae8-42ca-a473-5dcc9798e1e3" containerID="9b909cef1f3c5f14a2eb00a47c70765c32521f3265d248eeb3279f13c38c3fb7" exitCode=0
Sep 16 12:10:29.235030 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:29.234614    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps" event={"ID":"343ec258-cae8-42ca-a473-5dcc9798e1e3","Type":"ContainerDied","Data":"9b909cef1f3c5f14a2eb00a47c70765c32521f3265d248eeb3279f13c38c3fb7"}
Sep 16 12:10:32.248253 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:32.248217    2582 generic.go:356] "Generic (PLEG): container finished" podID="343ec258-cae8-42ca-a473-5dcc9798e1e3" containerID="7e5184aeb711c1f6b4ef74fc5649cde9387c4684aa4cef2d1e4d46713d828cf5" exitCode=0
Sep 16 12:10:32.248661 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:32.248268    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps" event={"ID":"343ec258-cae8-42ca-a473-5dcc9798e1e3","Type":"ContainerDied","Data":"7e5184aeb711c1f6b4ef74fc5649cde9387c4684aa4cef2d1e4d46713d828cf5"}
Sep 16 12:10:39.281382 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:39.281347    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps" event={"ID":"343ec258-cae8-42ca-a473-5dcc9798e1e3","Type":"ContainerStarted","Data":"2ec9a744ac8c1b9672a62a2aad8e0b9395ebebb62c3c7fb0a2daf10431f916aa"}
Sep 16 12:10:39.299469 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:39.299306    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps" podStartSLOduration=2.812763914 podStartE2EDuration="18.299286032s" totalImagesPullingTime="15.486522118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:10:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:10:39.297751741 +0000 UTC m=+598.548289288" watchObservedRunningTime="2026-09-16 12:10:39.299286032 +0000 UTC m=+598.549823582"
Sep 16 12:10:40.286706 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:40.286672    2582 generic.go:356] "Generic (PLEG): container finished" podID="343ec258-cae8-42ca-a473-5dcc9798e1e3" containerID="2ec9a744ac8c1b9672a62a2aad8e0b9395ebebb62c3c7fb0a2daf10431f916aa" exitCode=0
Sep 16 12:10:40.287153 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:40.286763    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps" event={"ID":"343ec258-cae8-42ca-a473-5dcc9798e1e3","Type":"ContainerDied","Data":"2ec9a744ac8c1b9672a62a2aad8e0b9395ebebb62c3c7fb0a2daf10431f916aa"}
Sep 16 12:10:41.432216 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:41.432192    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps"
Sep 16 12:10:41.489007 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:41.488972    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/343ec258-cae8-42ca-a473-5dcc9798e1e3-bundle\") pod \"343ec258-cae8-42ca-a473-5dcc9798e1e3\" (UID: \"343ec258-cae8-42ca-a473-5dcc9798e1e3\") "
Sep 16 12:10:41.489186 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:41.489020    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6tvh\" (UniqueName: \"kubernetes.io/projected/343ec258-cae8-42ca-a473-5dcc9798e1e3-kube-api-access-p6tvh\") pod \"343ec258-cae8-42ca-a473-5dcc9798e1e3\" (UID: \"343ec258-cae8-42ca-a473-5dcc9798e1e3\") "
Sep 16 12:10:41.489186 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:41.489065    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/343ec258-cae8-42ca-a473-5dcc9798e1e3-util\") pod \"343ec258-cae8-42ca-a473-5dcc9798e1e3\" (UID: \"343ec258-cae8-42ca-a473-5dcc9798e1e3\") "
Sep 16 12:10:41.489664 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:41.489636    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343ec258-cae8-42ca-a473-5dcc9798e1e3-bundle" (OuterVolumeSpecName: "bundle") pod "343ec258-cae8-42ca-a473-5dcc9798e1e3" (UID: "343ec258-cae8-42ca-a473-5dcc9798e1e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:10:41.491367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:41.491341    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343ec258-cae8-42ca-a473-5dcc9798e1e3-kube-api-access-p6tvh" (OuterVolumeSpecName: "kube-api-access-p6tvh") pod "343ec258-cae8-42ca-a473-5dcc9798e1e3" (UID: "343ec258-cae8-42ca-a473-5dcc9798e1e3"). InnerVolumeSpecName "kube-api-access-p6tvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:10:41.493699 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:41.493661    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343ec258-cae8-42ca-a473-5dcc9798e1e3-util" (OuterVolumeSpecName: "util") pod "343ec258-cae8-42ca-a473-5dcc9798e1e3" (UID: "343ec258-cae8-42ca-a473-5dcc9798e1e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:10:41.589983 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:41.589947    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/343ec258-cae8-42ca-a473-5dcc9798e1e3-util\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:10:41.589983 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:41.589979    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/343ec258-cae8-42ca-a473-5dcc9798e1e3-bundle\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:10:41.590182 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:41.589992    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6tvh\" (UniqueName: \"kubernetes.io/projected/343ec258-cae8-42ca-a473-5dcc9798e1e3-kube-api-access-p6tvh\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:10:42.296537 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:42.296496    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps" event={"ID":"343ec258-cae8-42ca-a473-5dcc9798e1e3","Type":"ContainerDied","Data":"c4c8028f40f0f50711930ec4b26ea06f8e51a5b9e740ecc44bc1de2563f93adb"}
Sep 16 12:10:42.297566 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:42.296555    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4c8028f40f0f50711930ec4b26ea06f8e51a5b9e740ecc44bc1de2563f93adb"
Sep 16 12:10:42.297566 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:10:42.296517    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wvbps"
Sep 16 12:11:00.251081 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.251039    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tfgxd"]
Sep 16 12:11:00.251682 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.251427    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="343ec258-cae8-42ca-a473-5dcc9798e1e3" containerName="util"
Sep 16 12:11:00.251682 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.251444    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="343ec258-cae8-42ca-a473-5dcc9798e1e3" containerName="util"
Sep 16 12:11:00.251682 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.251474    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="343ec258-cae8-42ca-a473-5dcc9798e1e3" containerName="extract"
Sep 16 12:11:00.251682 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.251482    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="343ec258-cae8-42ca-a473-5dcc9798e1e3" containerName="extract"
Sep 16 12:11:00.251682 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.251495    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="343ec258-cae8-42ca-a473-5dcc9798e1e3" containerName="pull"
Sep 16 12:11:00.251682 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.251503    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="343ec258-cae8-42ca-a473-5dcc9798e1e3" containerName="pull"
Sep 16 12:11:00.251682 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.251570    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="343ec258-cae8-42ca-a473-5dcc9798e1e3" containerName="extract"
Sep 16 12:11:00.255845 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.255810    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tfgxd"
Sep 16 12:11:00.258477 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.258457    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:11:00.258605 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.258588    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-wj46h\""
Sep 16 12:11:00.258650 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.258628    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 16 12:11:00.265647 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.265629    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tfgxd"]
Sep 16 12:11:00.333231 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.333190    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wqbr\" (UniqueName: \"kubernetes.io/projected/4b14c7bd-082d-4da8-a002-5043fc8fb9b7-kube-api-access-5wqbr\") pod \"cert-manager-operator-controller-manager-55c444bf86-tfgxd\" (UID: \"4b14c7bd-082d-4da8-a002-5043fc8fb9b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tfgxd"
Sep 16 12:11:00.333422 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.333244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b14c7bd-082d-4da8-a002-5043fc8fb9b7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tfgxd\" (UID: \"4b14c7bd-082d-4da8-a002-5043fc8fb9b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tfgxd"
Sep 16 12:11:00.434774 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.434094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wqbr\" (UniqueName: \"kubernetes.io/projected/4b14c7bd-082d-4da8-a002-5043fc8fb9b7-kube-api-access-5wqbr\") pod \"cert-manager-operator-controller-manager-55c444bf86-tfgxd\" (UID: \"4b14c7bd-082d-4da8-a002-5043fc8fb9b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tfgxd"
Sep 16 12:11:00.434774 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.434141    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b14c7bd-082d-4da8-a002-5043fc8fb9b7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tfgxd\" (UID: \"4b14c7bd-082d-4da8-a002-5043fc8fb9b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tfgxd"
Sep 16 12:11:00.434774 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.434596    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b14c7bd-082d-4da8-a002-5043fc8fb9b7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tfgxd\" (UID: \"4b14c7bd-082d-4da8-a002-5043fc8fb9b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tfgxd"
Sep 16 12:11:00.446223 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.446196    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wqbr\" (UniqueName: \"kubernetes.io/projected/4b14c7bd-082d-4da8-a002-5043fc8fb9b7-kube-api-access-5wqbr\") pod \"cert-manager-operator-controller-manager-55c444bf86-tfgxd\" (UID: \"4b14c7bd-082d-4da8-a002-5043fc8fb9b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tfgxd"
Sep 16 12:11:00.565404 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.565320    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tfgxd"
Sep 16 12:11:00.853460 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.853433    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tfgxd"]
Sep 16 12:11:00.855097 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:11:00.855074    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b14c7bd_082d_4da8_a002_5043fc8fb9b7.slice/crio-0cb08ff43a14e819fd1272d9b4a3dbcd751125abfa2a10cc79d1a0a4024ce24e WatchSource:0}: Error finding container 0cb08ff43a14e819fd1272d9b4a3dbcd751125abfa2a10cc79d1a0a4024ce24e: Status 404 returned error can't find the container with id 0cb08ff43a14e819fd1272d9b4a3dbcd751125abfa2a10cc79d1a0a4024ce24e
Sep 16 12:11:00.857762 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:00.857726    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:11:01.371526 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:01.371486    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tfgxd" event={"ID":"4b14c7bd-082d-4da8-a002-5043fc8fb9b7","Type":"ContainerStarted","Data":"0cb08ff43a14e819fd1272d9b4a3dbcd751125abfa2a10cc79d1a0a4024ce24e"}
Sep 16 12:11:05.390588 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:05.390540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tfgxd" event={"ID":"4b14c7bd-082d-4da8-a002-5043fc8fb9b7","Type":"ContainerStarted","Data":"96c5cc965268250343cc49a18eb7eff72a90236eed59d98da3264c4a18a56697"}
Sep 16 12:11:05.413714 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:05.413656    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tfgxd" podStartSLOduration=1.469974074 podStartE2EDuration="5.413639474s" totalImagesPullingTime="3.9436654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:11:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:11:05.413460592 +0000 UTC m=+624.663998141" watchObservedRunningTime="2026-09-16 12:11:05.413639474 +0000 UTC m=+624.664177023"
Sep 16 12:11:05.849253 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:05.849219    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jbfck"]
Sep 16 12:11:05.851485 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:05.851465    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jbfck"
Sep 16 12:11:05.855004 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:05.854972    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-9t898\""
Sep 16 12:11:05.855128 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:05.855053    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 12:11:05.856175 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:05.856154    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 12:11:05.866009 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:05.865983    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jbfck"]
Sep 16 12:11:05.981905 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:05.981869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5rpk\" (UniqueName: \"kubernetes.io/projected/88012815-cbd5-4375-9337-f94f1bf3119c-kube-api-access-r5rpk\") pod \"cert-manager-webhook-6486ffd997-jbfck\" (UID: \"88012815-cbd5-4375-9337-f94f1bf3119c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jbfck"
Sep 16 12:11:05.982102 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:05.981928    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88012815-cbd5-4375-9337-f94f1bf3119c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jbfck\" (UID: \"88012815-cbd5-4375-9337-f94f1bf3119c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jbfck"
Sep 16 12:11:06.082623 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:06.082586    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5rpk\" (UniqueName: \"kubernetes.io/projected/88012815-cbd5-4375-9337-f94f1bf3119c-kube-api-access-r5rpk\") pod \"cert-manager-webhook-6486ffd997-jbfck\" (UID: \"88012815-cbd5-4375-9337-f94f1bf3119c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jbfck"
Sep 16 12:11:06.082844 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:06.082648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88012815-cbd5-4375-9337-f94f1bf3119c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jbfck\" (UID: \"88012815-cbd5-4375-9337-f94f1bf3119c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jbfck"
Sep 16 12:11:06.101960 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:06.101896    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88012815-cbd5-4375-9337-f94f1bf3119c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jbfck\" (UID: \"88012815-cbd5-4375-9337-f94f1bf3119c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jbfck"
Sep 16 12:11:06.101960 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:06.101896    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5rpk\" (UniqueName: \"kubernetes.io/projected/88012815-cbd5-4375-9337-f94f1bf3119c-kube-api-access-r5rpk\") pod \"cert-manager-webhook-6486ffd997-jbfck\" (UID: \"88012815-cbd5-4375-9337-f94f1bf3119c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jbfck"
Sep 16 12:11:06.177465 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:06.177431    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jbfck"
Sep 16 12:11:06.469771 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:06.469545    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jbfck"]
Sep 16 12:11:06.472186 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:11:06.472158    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88012815_cbd5_4375_9337_f94f1bf3119c.slice/crio-abc016aa08255dd33a867766eca32f7404668cec62c0666e493f038349bdb9e0 WatchSource:0}: Error finding container abc016aa08255dd33a867766eca32f7404668cec62c0666e493f038349bdb9e0: Status 404 returned error can't find the container with id abc016aa08255dd33a867766eca32f7404668cec62c0666e493f038349bdb9e0
Sep 16 12:11:07.400431 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:07.400390    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jbfck" event={"ID":"88012815-cbd5-4375-9337-f94f1bf3119c","Type":"ContainerStarted","Data":"abc016aa08255dd33a867766eca32f7404668cec62c0666e493f038349bdb9e0"}
Sep 16 12:11:07.449024 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:07.448989    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4"]
Sep 16 12:11:07.451898 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:07.451881    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4"
Sep 16 12:11:07.454345 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:07.454322    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bp9w9\""
Sep 16 12:11:07.454476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:07.454349    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:11:07.454476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:07.454324    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:11:07.460042 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:07.460016    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4"]
Sep 16 12:11:07.495826 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:07.495773    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fdba4fd-7988-47db-a86f-e85a54836487-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4\" (UID: \"8fdba4fd-7988-47db-a86f-e85a54836487\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4"
Sep 16 12:11:07.495826 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:07.495834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fdba4fd-7988-47db-a86f-e85a54836487-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4\" (UID: \"8fdba4fd-7988-47db-a86f-e85a54836487\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4"
Sep 16 12:11:07.496241 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:07.495872    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9j9s\" (UniqueName: \"kubernetes.io/projected/8fdba4fd-7988-47db-a86f-e85a54836487-kube-api-access-v9j9s\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4\" (UID: \"8fdba4fd-7988-47db-a86f-e85a54836487\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4"
Sep 16 12:11:07.596549 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:07.596514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fdba4fd-7988-47db-a86f-e85a54836487-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4\" (UID: \"8fdba4fd-7988-47db-a86f-e85a54836487\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4"
Sep 16 12:11:07.596549 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:07.596552    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fdba4fd-7988-47db-a86f-e85a54836487-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4\" (UID: \"8fdba4fd-7988-47db-a86f-e85a54836487\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4"
Sep 16 12:11:07.596798 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:07.596575    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9j9s\" (UniqueName: \"kubernetes.io/projected/8fdba4fd-7988-47db-a86f-e85a54836487-kube-api-access-v9j9s\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4\" (UID: \"8fdba4fd-7988-47db-a86f-e85a54836487\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4"
Sep 16 12:11:07.597043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:07.597020    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fdba4fd-7988-47db-a86f-e85a54836487-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4\" (UID: \"8fdba4fd-7988-47db-a86f-e85a54836487\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4"
Sep 16 12:11:07.597096 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:07.597080    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fdba4fd-7988-47db-a86f-e85a54836487-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4\" (UID: \"8fdba4fd-7988-47db-a86f-e85a54836487\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4"
Sep 16 12:11:07.605316 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:07.605282    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9j9s\" (UniqueName: \"kubernetes.io/projected/8fdba4fd-7988-47db-a86f-e85a54836487-kube-api-access-v9j9s\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4\" (UID: \"8fdba4fd-7988-47db-a86f-e85a54836487\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4"
Sep 16 12:11:07.762502 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:07.762419    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4"
Sep 16 12:11:08.043667 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:08.043641    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4"]
Sep 16 12:11:08.046017 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:11:08.045987    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fdba4fd_7988_47db_a86f_e85a54836487.slice/crio-2a8289d2c83b7c78bdfa46fd748fbe1bebeb4cbac18677be277100929fa62b65 WatchSource:0}: Error finding container 2a8289d2c83b7c78bdfa46fd748fbe1bebeb4cbac18677be277100929fa62b65: Status 404 returned error can't find the container with id 2a8289d2c83b7c78bdfa46fd748fbe1bebeb4cbac18677be277100929fa62b65
Sep 16 12:11:08.213503 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:08.213473    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vd6cl"]
Sep 16 12:11:08.216179 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:08.216140    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vd6cl"
Sep 16 12:11:08.227627 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:08.227598    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vd6cl"]
Sep 16 12:11:08.302109 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:08.302004    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0133753b-8bea-4429-b53a-145592eba6ec-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vd6cl\" (UID: \"0133753b-8bea-4429-b53a-145592eba6ec\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vd6cl"
Sep 16 12:11:08.302109 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:08.302064    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86wd4\" (UniqueName: \"kubernetes.io/projected/0133753b-8bea-4429-b53a-145592eba6ec-kube-api-access-86wd4\") pod \"cert-manager-cainjector-559cdd8c87-vd6cl\" (UID: \"0133753b-8bea-4429-b53a-145592eba6ec\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vd6cl"
Sep 16 12:11:08.402499 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:08.402458    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0133753b-8bea-4429-b53a-145592eba6ec-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vd6cl\" (UID: \"0133753b-8bea-4429-b53a-145592eba6ec\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vd6cl"
Sep 16 12:11:08.402499 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:08.402504    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86wd4\" (UniqueName: \"kubernetes.io/projected/0133753b-8bea-4429-b53a-145592eba6ec-kube-api-access-86wd4\") pod \"cert-manager-cainjector-559cdd8c87-vd6cl\" (UID: \"0133753b-8bea-4429-b53a-145592eba6ec\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vd6cl"
Sep 16 12:11:08.407013 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:08.406980    2582 generic.go:356] "Generic (PLEG): container finished" podID="8fdba4fd-7988-47db-a86f-e85a54836487" containerID="17773fa1bc0e9b0655369c610659063710b6984c534dbd17d6e0d4c7b93cf175" exitCode=0
Sep 16 12:11:08.407181 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:08.407046    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4" event={"ID":"8fdba4fd-7988-47db-a86f-e85a54836487","Type":"ContainerDied","Data":"17773fa1bc0e9b0655369c610659063710b6984c534dbd17d6e0d4c7b93cf175"}
Sep 16 12:11:08.407181 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:08.407137    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4" event={"ID":"8fdba4fd-7988-47db-a86f-e85a54836487","Type":"ContainerStarted","Data":"2a8289d2c83b7c78bdfa46fd748fbe1bebeb4cbac18677be277100929fa62b65"}
Sep 16 12:11:08.415544 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:08.415513    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86wd4\" (UniqueName: \"kubernetes.io/projected/0133753b-8bea-4429-b53a-145592eba6ec-kube-api-access-86wd4\") pod \"cert-manager-cainjector-559cdd8c87-vd6cl\" (UID: \"0133753b-8bea-4429-b53a-145592eba6ec\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vd6cl"
Sep 16 12:11:08.415835 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:08.415811    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0133753b-8bea-4429-b53a-145592eba6ec-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vd6cl\" (UID: \"0133753b-8bea-4429-b53a-145592eba6ec\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vd6cl"
Sep 16 12:11:08.535148 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:08.535114    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vd6cl"
Sep 16 12:11:08.818315 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:08.818017    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vd6cl"]
Sep 16 12:11:08.821267 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:11:08.821233    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0133753b_8bea_4429_b53a_145592eba6ec.slice/crio-d24f75e64754b3fef4ea71447bdc744264252ac224e4a36de53287709077084f WatchSource:0}: Error finding container d24f75e64754b3fef4ea71447bdc744264252ac224e4a36de53287709077084f: Status 404 returned error can't find the container with id d24f75e64754b3fef4ea71447bdc744264252ac224e4a36de53287709077084f
Sep 16 12:11:09.414344 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:09.414310    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vd6cl" event={"ID":"0133753b-8bea-4429-b53a-145592eba6ec","Type":"ContainerStarted","Data":"d24f75e64754b3fef4ea71447bdc744264252ac224e4a36de53287709077084f"}
Sep 16 12:11:10.421817 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:10.421774    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vd6cl" event={"ID":"0133753b-8bea-4429-b53a-145592eba6ec","Type":"ContainerStarted","Data":"ab159bf45dba22c4f02d5d1606900798c2f6e4f6b837bd6b5360f0dafd78adc9"}
Sep 16 12:11:10.423624 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:10.423589    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jbfck" event={"ID":"88012815-cbd5-4375-9337-f94f1bf3119c","Type":"ContainerStarted","Data":"39f6d387dbb9ca8c5d0b95d321d6149061ba3b36b5e1c4aa91760fd32b5671ec"}
Sep 16 12:11:10.423775 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:10.423756    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jbfck"
Sep 16 12:11:10.438246 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:10.438202    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vd6cl" podStartSLOduration=1.485403829 podStartE2EDuration="2.438187389s" totalImagesPullingTime="952.78356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:11:10.436312919 +0000 UTC m=+629.686850469" watchObservedRunningTime="2026-09-16 12:11:10.438187389 +0000 UTC m=+629.688724939"
Sep 16 12:11:10.451860 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:10.451798    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-jbfck" podStartSLOduration=2.149914183 podStartE2EDuration="5.451768678s" totalImagesPullingTime="3.301854495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:11:10.451072479 +0000 UTC m=+629.701610029" watchObservedRunningTime="2026-09-16 12:11:10.451768678 +0000 UTC m=+629.702306227"
Sep 16 12:11:11.429901 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:11.429798    2582 generic.go:356] "Generic (PLEG): container finished" podID="8fdba4fd-7988-47db-a86f-e85a54836487" containerID="98e296de745ce54d46fcfb37ef5ed1e4e7a4aa793c39e467ffa4da74fa31069b" exitCode=0
Sep 16 12:11:11.430375 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:11.429898    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4" event={"ID":"8fdba4fd-7988-47db-a86f-e85a54836487","Type":"ContainerDied","Data":"98e296de745ce54d46fcfb37ef5ed1e4e7a4aa793c39e467ffa4da74fa31069b"}
Sep 16 12:11:12.436616 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:12.436582    2582 generic.go:356] "Generic (PLEG): container finished" podID="8fdba4fd-7988-47db-a86f-e85a54836487" containerID="fae3b1b4697ef1553490cc399affe7e6335f6f9d2a57c05e1d1844b1384fbdf6" exitCode=0
Sep 16 12:11:12.437093 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:12.436673    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4" event={"ID":"8fdba4fd-7988-47db-a86f-e85a54836487","Type":"ContainerDied","Data":"fae3b1b4697ef1553490cc399affe7e6335f6f9d2a57c05e1d1844b1384fbdf6"}
Sep 16 12:11:13.565427 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:13.565397    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4"
Sep 16 12:11:13.645917 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:13.645876    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fdba4fd-7988-47db-a86f-e85a54836487-bundle\") pod \"8fdba4fd-7988-47db-a86f-e85a54836487\" (UID: \"8fdba4fd-7988-47db-a86f-e85a54836487\") "
Sep 16 12:11:13.646091 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:13.645945    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9j9s\" (UniqueName: \"kubernetes.io/projected/8fdba4fd-7988-47db-a86f-e85a54836487-kube-api-access-v9j9s\") pod \"8fdba4fd-7988-47db-a86f-e85a54836487\" (UID: \"8fdba4fd-7988-47db-a86f-e85a54836487\") "
Sep 16 12:11:13.646091 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:13.645999    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fdba4fd-7988-47db-a86f-e85a54836487-util\") pod \"8fdba4fd-7988-47db-a86f-e85a54836487\" (UID: \"8fdba4fd-7988-47db-a86f-e85a54836487\") "
Sep 16 12:11:13.646384 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:13.646357    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdba4fd-7988-47db-a86f-e85a54836487-bundle" (OuterVolumeSpecName: "bundle") pod "8fdba4fd-7988-47db-a86f-e85a54836487" (UID: "8fdba4fd-7988-47db-a86f-e85a54836487"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:11:13.648346 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:13.648323    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fdba4fd-7988-47db-a86f-e85a54836487-kube-api-access-v9j9s" (OuterVolumeSpecName: "kube-api-access-v9j9s") pod "8fdba4fd-7988-47db-a86f-e85a54836487" (UID: "8fdba4fd-7988-47db-a86f-e85a54836487"). InnerVolumeSpecName "kube-api-access-v9j9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:11:13.650376 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:13.650337    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdba4fd-7988-47db-a86f-e85a54836487-util" (OuterVolumeSpecName: "util") pod "8fdba4fd-7988-47db-a86f-e85a54836487" (UID: "8fdba4fd-7988-47db-a86f-e85a54836487"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:11:13.747048 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:13.746963    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fdba4fd-7988-47db-a86f-e85a54836487-util\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:11:13.747048 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:13.746990    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fdba4fd-7988-47db-a86f-e85a54836487-bundle\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:11:13.747048 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:13.746999    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9j9s\" (UniqueName: \"kubernetes.io/projected/8fdba4fd-7988-47db-a86f-e85a54836487-kube-api-access-v9j9s\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:11:14.446699 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:14.446665    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4" event={"ID":"8fdba4fd-7988-47db-a86f-e85a54836487","Type":"ContainerDied","Data":"2a8289d2c83b7c78bdfa46fd748fbe1bebeb4cbac18677be277100929fa62b65"}
Sep 16 12:11:14.446699 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:14.446691    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdlwf4"
Sep 16 12:11:14.446699 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:14.446710    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a8289d2c83b7c78bdfa46fd748fbe1bebeb4cbac18677be277100929fa62b65"
Sep 16 12:11:16.433135 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:16.433106    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jbfck"
Sep 16 12:11:17.760575 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:17.760545    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nkrbp"]
Sep 16 12:11:17.761036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:17.760844    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fdba4fd-7988-47db-a86f-e85a54836487" containerName="util"
Sep 16 12:11:17.761036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:17.760855    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fdba4fd-7988-47db-a86f-e85a54836487" containerName="util"
Sep 16 12:11:17.761036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:17.760869    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fdba4fd-7988-47db-a86f-e85a54836487" containerName="pull"
Sep 16 12:11:17.761036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:17.760874    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fdba4fd-7988-47db-a86f-e85a54836487" containerName="pull"
Sep 16 12:11:17.761036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:17.760884    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fdba4fd-7988-47db-a86f-e85a54836487" containerName="extract"
Sep 16 12:11:17.761036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:17.760889    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fdba4fd-7988-47db-a86f-e85a54836487" containerName="extract"
Sep 16 12:11:17.761036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:17.760937    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fdba4fd-7988-47db-a86f-e85a54836487" containerName="extract"
Sep 16 12:11:17.766332 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:17.766313    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nkrbp"
Sep 16 12:11:17.769012 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:17.768987    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jq6v4\""
Sep 16 12:11:17.773017 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:17.772994    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nkrbp"]
Sep 16 12:11:17.880839 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:17.880798    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9b5c451-436d-4708-ae63-7f9674441bd8-bound-sa-token\") pod \"cert-manager-55b4d954d5-nkrbp\" (UID: \"b9b5c451-436d-4708-ae63-7f9674441bd8\") " pod="cert-manager/cert-manager-55b4d954d5-nkrbp"
Sep 16 12:11:17.880839 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:17.880838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kckv2\" (UniqueName: \"kubernetes.io/projected/b9b5c451-436d-4708-ae63-7f9674441bd8-kube-api-access-kckv2\") pod \"cert-manager-55b4d954d5-nkrbp\" (UID: \"b9b5c451-436d-4708-ae63-7f9674441bd8\") " pod="cert-manager/cert-manager-55b4d954d5-nkrbp"
Sep 16 12:11:17.981917 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:17.981877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9b5c451-436d-4708-ae63-7f9674441bd8-bound-sa-token\") pod \"cert-manager-55b4d954d5-nkrbp\" (UID: \"b9b5c451-436d-4708-ae63-7f9674441bd8\") " pod="cert-manager/cert-manager-55b4d954d5-nkrbp"
Sep 16 12:11:17.982065 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:17.981925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kckv2\" (UniqueName: \"kubernetes.io/projected/b9b5c451-436d-4708-ae63-7f9674441bd8-kube-api-access-kckv2\") pod \"cert-manager-55b4d954d5-nkrbp\" (UID: \"b9b5c451-436d-4708-ae63-7f9674441bd8\") " pod="cert-manager/cert-manager-55b4d954d5-nkrbp"
Sep 16 12:11:17.990712 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:17.990678    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9b5c451-436d-4708-ae63-7f9674441bd8-bound-sa-token\") pod \"cert-manager-55b4d954d5-nkrbp\" (UID: \"b9b5c451-436d-4708-ae63-7f9674441bd8\") " pod="cert-manager/cert-manager-55b4d954d5-nkrbp"
Sep 16 12:11:17.990712 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:17.990706    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kckv2\" (UniqueName: \"kubernetes.io/projected/b9b5c451-436d-4708-ae63-7f9674441bd8-kube-api-access-kckv2\") pod \"cert-manager-55b4d954d5-nkrbp\" (UID: \"b9b5c451-436d-4708-ae63-7f9674441bd8\") " pod="cert-manager/cert-manager-55b4d954d5-nkrbp"
Sep 16 12:11:18.076462 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:18.076424    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nkrbp"
Sep 16 12:11:18.346599 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:18.346574    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nkrbp"]
Sep 16 12:11:18.349522 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:11:18.349475    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9b5c451_436d_4708_ae63_7f9674441bd8.slice/crio-2ae46f37b5fdd8cb675e3cd0edc37d0dc6381b4acd487bcacf1d47f61c135a3e WatchSource:0}: Error finding container 2ae46f37b5fdd8cb675e3cd0edc37d0dc6381b4acd487bcacf1d47f61c135a3e: Status 404 returned error can't find the container with id 2ae46f37b5fdd8cb675e3cd0edc37d0dc6381b4acd487bcacf1d47f61c135a3e
Sep 16 12:11:18.464034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:18.463998    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nkrbp" event={"ID":"b9b5c451-436d-4708-ae63-7f9674441bd8","Type":"ContainerStarted","Data":"e81ecaf941413a8fabf49963618b79716dfd84024fe2d869dc8083456f13789f"}
Sep 16 12:11:18.464034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:18.464044    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nkrbp" event={"ID":"b9b5c451-436d-4708-ae63-7f9674441bd8","Type":"ContainerStarted","Data":"2ae46f37b5fdd8cb675e3cd0edc37d0dc6381b4acd487bcacf1d47f61c135a3e"}
Sep 16 12:11:18.482142 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:18.482096    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-nkrbp" podStartSLOduration=1.482081084 podStartE2EDuration="1.482081084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:11:18.479595705 +0000 UTC m=+637.730133279" watchObservedRunningTime="2026-09-16 12:11:18.482081084 +0000 UTC m=+637.732618633"
Sep 16 12:11:27.337411 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:27.337375    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5"]
Sep 16 12:11:27.341843 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:27.341825    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5"
Sep 16 12:11:27.344703 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:27.344673    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:11:27.344830 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:27.344755    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bp9w9\""
Sep 16 12:11:27.345609 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:27.345592    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:11:27.350849 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:27.350824    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5"]
Sep 16 12:11:27.467103 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:27.467069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5\" (UID: \"f6ea4e79-aab0-4aa7-9688-50c15b07f5be\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5"
Sep 16 12:11:27.467283 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:27.467120    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5\" (UID: \"f6ea4e79-aab0-4aa7-9688-50c15b07f5be\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5"
Sep 16 12:11:27.467283 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:27.467180    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gz9s\" (UniqueName: \"kubernetes.io/projected/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-kube-api-access-2gz9s\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5\" (UID: \"f6ea4e79-aab0-4aa7-9688-50c15b07f5be\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5"
Sep 16 12:11:27.568456 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:27.568422    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gz9s\" (UniqueName: \"kubernetes.io/projected/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-kube-api-access-2gz9s\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5\" (UID: \"f6ea4e79-aab0-4aa7-9688-50c15b07f5be\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5"
Sep 16 12:11:27.568659 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:27.568467    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5\" (UID: \"f6ea4e79-aab0-4aa7-9688-50c15b07f5be\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5"
Sep 16 12:11:27.568659 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:27.568591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5\" (UID: \"f6ea4e79-aab0-4aa7-9688-50c15b07f5be\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5"
Sep 16 12:11:27.568905 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:27.568885    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5\" (UID: \"f6ea4e79-aab0-4aa7-9688-50c15b07f5be\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5"
Sep 16 12:11:27.568961 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:27.568922    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5\" (UID: \"f6ea4e79-aab0-4aa7-9688-50c15b07f5be\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5"
Sep 16 12:11:27.576334 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:27.576313    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gz9s\" (UniqueName: \"kubernetes.io/projected/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-kube-api-access-2gz9s\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5\" (UID: \"f6ea4e79-aab0-4aa7-9688-50c15b07f5be\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5"
Sep 16 12:11:27.651578 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:27.651482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5"
Sep 16 12:11:27.922973 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:27.922946    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5"]
Sep 16 12:11:27.925425 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:11:27.925395    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ea4e79_aab0_4aa7_9688_50c15b07f5be.slice/crio-4309a610455833f4b37ec714741bdf81e5d35b506864e32659fa4d1de9e08866 WatchSource:0}: Error finding container 4309a610455833f4b37ec714741bdf81e5d35b506864e32659fa4d1de9e08866: Status 404 returned error can't find the container with id 4309a610455833f4b37ec714741bdf81e5d35b506864e32659fa4d1de9e08866
Sep 16 12:11:28.510504 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:28.510469    2582 generic.go:356] "Generic (PLEG): container finished" podID="f6ea4e79-aab0-4aa7-9688-50c15b07f5be" containerID="c260bd1ef2b1056d1da498e689a9a038558a4c9badb44e7ecb7823e10bd9f941" exitCode=0
Sep 16 12:11:28.511002 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:28.510547    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5" event={"ID":"f6ea4e79-aab0-4aa7-9688-50c15b07f5be","Type":"ContainerDied","Data":"c260bd1ef2b1056d1da498e689a9a038558a4c9badb44e7ecb7823e10bd9f941"}
Sep 16 12:11:28.511002 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:28.510621    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5" event={"ID":"f6ea4e79-aab0-4aa7-9688-50c15b07f5be","Type":"ContainerStarted","Data":"4309a610455833f4b37ec714741bdf81e5d35b506864e32659fa4d1de9e08866"}
Sep 16 12:11:29.517053 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:29.516952    2582 generic.go:356] "Generic (PLEG): container finished" podID="f6ea4e79-aab0-4aa7-9688-50c15b07f5be" containerID="fd26a54a43328244d241efd82cebd3b2f4f1dee813be1249a4cf8a2727a5c87a" exitCode=0
Sep 16 12:11:29.517053 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:29.517005    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5" event={"ID":"f6ea4e79-aab0-4aa7-9688-50c15b07f5be","Type":"ContainerDied","Data":"fd26a54a43328244d241efd82cebd3b2f4f1dee813be1249a4cf8a2727a5c87a"}
Sep 16 12:11:30.524033 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:30.523994    2582 generic.go:356] "Generic (PLEG): container finished" podID="f6ea4e79-aab0-4aa7-9688-50c15b07f5be" containerID="33267e8f6e91a83d89a0b9a32f06bd458f7d217f640283d75bc1d28c987b2e04" exitCode=0
Sep 16 12:11:30.524452 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:30.524065    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5" event={"ID":"f6ea4e79-aab0-4aa7-9688-50c15b07f5be","Type":"ContainerDied","Data":"33267e8f6e91a83d89a0b9a32f06bd458f7d217f640283d75bc1d28c987b2e04"}
Sep 16 12:11:31.655162 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:31.655139    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5"
Sep 16 12:11:31.800469 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:31.800368    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-bundle\") pod \"f6ea4e79-aab0-4aa7-9688-50c15b07f5be\" (UID: \"f6ea4e79-aab0-4aa7-9688-50c15b07f5be\") "
Sep 16 12:11:31.800469 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:31.800440    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gz9s\" (UniqueName: \"kubernetes.io/projected/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-kube-api-access-2gz9s\") pod \"f6ea4e79-aab0-4aa7-9688-50c15b07f5be\" (UID: \"f6ea4e79-aab0-4aa7-9688-50c15b07f5be\") "
Sep 16 12:11:31.800469 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:31.800468    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-util\") pod \"f6ea4e79-aab0-4aa7-9688-50c15b07f5be\" (UID: \"f6ea4e79-aab0-4aa7-9688-50c15b07f5be\") "
Sep 16 12:11:31.801296 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:31.801269    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-bundle" (OuterVolumeSpecName: "bundle") pod "f6ea4e79-aab0-4aa7-9688-50c15b07f5be" (UID: "f6ea4e79-aab0-4aa7-9688-50c15b07f5be"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:11:31.802746 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:31.802711    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-kube-api-access-2gz9s" (OuterVolumeSpecName: "kube-api-access-2gz9s") pod "f6ea4e79-aab0-4aa7-9688-50c15b07f5be" (UID: "f6ea4e79-aab0-4aa7-9688-50c15b07f5be"). InnerVolumeSpecName "kube-api-access-2gz9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:11:31.808930 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:31.808897    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-util" (OuterVolumeSpecName: "util") pod "f6ea4e79-aab0-4aa7-9688-50c15b07f5be" (UID: "f6ea4e79-aab0-4aa7-9688-50c15b07f5be"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:11:31.901305 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:31.901267    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gz9s\" (UniqueName: \"kubernetes.io/projected/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-kube-api-access-2gz9s\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:11:31.901305 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:31.901295    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-util\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:11:31.901305 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:31.901305    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6ea4e79-aab0-4aa7-9688-50c15b07f5be-bundle\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:11:32.535047 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:32.535013    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5" event={"ID":"f6ea4e79-aab0-4aa7-9688-50c15b07f5be","Type":"ContainerDied","Data":"4309a610455833f4b37ec714741bdf81e5d35b506864e32659fa4d1de9e08866"}
Sep 16 12:11:32.535210 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:32.535066    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4309a610455833f4b37ec714741bdf81e5d35b506864e32659fa4d1de9e08866"
Sep 16 12:11:32.535210 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:32.535026    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbx4b5"
Sep 16 12:11:44.195995 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.195958    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"]
Sep 16 12:11:44.196380 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.196271    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6ea4e79-aab0-4aa7-9688-50c15b07f5be" containerName="extract"
Sep 16 12:11:44.196380 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.196283    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6ea4e79-aab0-4aa7-9688-50c15b07f5be" containerName="extract"
Sep 16 12:11:44.196380 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.196292    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6ea4e79-aab0-4aa7-9688-50c15b07f5be" containerName="util"
Sep 16 12:11:44.196380 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.196297    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6ea4e79-aab0-4aa7-9688-50c15b07f5be" containerName="util"
Sep 16 12:11:44.196380 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.196305    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6ea4e79-aab0-4aa7-9688-50c15b07f5be" containerName="pull"
Sep 16 12:11:44.196380 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.196311    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6ea4e79-aab0-4aa7-9688-50c15b07f5be" containerName="pull"
Sep 16 12:11:44.196380 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.196366    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6ea4e79-aab0-4aa7-9688-50c15b07f5be" containerName="extract"
Sep 16 12:11:44.199594 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.199575    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:44.203507 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.203479    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 12:11:44.203651 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.203577    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lcl5n\""
Sep 16 12:11:44.203651 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.203605    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 12:11:44.203837 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.203655    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 12:11:44.203837 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.203767    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 12:11:44.242483 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.242453    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"]
Sep 16 12:11:44.295497 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.295467    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxs7\" (UniqueName: \"kubernetes.io/projected/5eed7827-c019-4b9d-b655-9c1c36223c1d-kube-api-access-bnxs7\") pod \"opendatahub-operator-controller-manager-7887d7f7b8-6v6jp\" (UID: \"5eed7827-c019-4b9d-b655-9c1c36223c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:44.295728 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.295524    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5eed7827-c019-4b9d-b655-9c1c36223c1d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7887d7f7b8-6v6jp\" (UID: \"5eed7827-c019-4b9d-b655-9c1c36223c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:44.295728 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.295608    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5eed7827-c019-4b9d-b655-9c1c36223c1d-manifests\") pod \"opendatahub-operator-controller-manager-7887d7f7b8-6v6jp\" (UID: \"5eed7827-c019-4b9d-b655-9c1c36223c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:44.295728 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.295654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5eed7827-c019-4b9d-b655-9c1c36223c1d-tmp\") pod \"opendatahub-operator-controller-manager-7887d7f7b8-6v6jp\" (UID: \"5eed7827-c019-4b9d-b655-9c1c36223c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:44.295728 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.295695    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5eed7827-c019-4b9d-b655-9c1c36223c1d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7887d7f7b8-6v6jp\" (UID: \"5eed7827-c019-4b9d-b655-9c1c36223c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:44.358762 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.358703    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4"]
Sep 16 12:11:44.362296 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.362277    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4"
Sep 16 12:11:44.364868 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.364845    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:11:44.364981 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.364844    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:11:44.364981 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.364910    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bp9w9\""
Sep 16 12:11:44.371220 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.371197    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4"]
Sep 16 12:11:44.396187 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.396159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5eed7827-c019-4b9d-b655-9c1c36223c1d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7887d7f7b8-6v6jp\" (UID: \"5eed7827-c019-4b9d-b655-9c1c36223c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:44.396361 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.396199    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc22k\" (UniqueName: \"kubernetes.io/projected/641bf342-016f-4d7e-a14c-1f5abdafceb3-kube-api-access-xc22k\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4\" (UID: \"641bf342-016f-4d7e-a14c-1f5abdafceb3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4"
Sep 16 12:11:44.396361 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.396230    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5eed7827-c019-4b9d-b655-9c1c36223c1d-manifests\") pod \"opendatahub-operator-controller-manager-7887d7f7b8-6v6jp\" (UID: \"5eed7827-c019-4b9d-b655-9c1c36223c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:44.396361 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.396248    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/641bf342-016f-4d7e-a14c-1f5abdafceb3-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4\" (UID: \"641bf342-016f-4d7e-a14c-1f5abdafceb3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4"
Sep 16 12:11:44.396361 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.396272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5eed7827-c019-4b9d-b655-9c1c36223c1d-tmp\") pod \"opendatahub-operator-controller-manager-7887d7f7b8-6v6jp\" (UID: \"5eed7827-c019-4b9d-b655-9c1c36223c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:44.396361 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.396288    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5eed7827-c019-4b9d-b655-9c1c36223c1d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7887d7f7b8-6v6jp\" (UID: \"5eed7827-c019-4b9d-b655-9c1c36223c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:44.396639 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.396377    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxs7\" (UniqueName: \"kubernetes.io/projected/5eed7827-c019-4b9d-b655-9c1c36223c1d-kube-api-access-bnxs7\") pod \"opendatahub-operator-controller-manager-7887d7f7b8-6v6jp\" (UID: \"5eed7827-c019-4b9d-b655-9c1c36223c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:44.396639 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.396435    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/641bf342-016f-4d7e-a14c-1f5abdafceb3-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4\" (UID: \"641bf342-016f-4d7e-a14c-1f5abdafceb3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4"
Sep 16 12:11:44.396730 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.396676    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5eed7827-c019-4b9d-b655-9c1c36223c1d-manifests\") pod \"opendatahub-operator-controller-manager-7887d7f7b8-6v6jp\" (UID: \"5eed7827-c019-4b9d-b655-9c1c36223c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:44.398784 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.398763    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5eed7827-c019-4b9d-b655-9c1c36223c1d-tmp\") pod \"opendatahub-operator-controller-manager-7887d7f7b8-6v6jp\" (UID: \"5eed7827-c019-4b9d-b655-9c1c36223c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:44.399036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.399014    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5eed7827-c019-4b9d-b655-9c1c36223c1d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7887d7f7b8-6v6jp\" (UID: \"5eed7827-c019-4b9d-b655-9c1c36223c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:44.399097 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.399046    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5eed7827-c019-4b9d-b655-9c1c36223c1d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7887d7f7b8-6v6jp\" (UID: \"5eed7827-c019-4b9d-b655-9c1c36223c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:44.404609 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.404585    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxs7\" (UniqueName: \"kubernetes.io/projected/5eed7827-c019-4b9d-b655-9c1c36223c1d-kube-api-access-bnxs7\") pod \"opendatahub-operator-controller-manager-7887d7f7b8-6v6jp\" (UID: \"5eed7827-c019-4b9d-b655-9c1c36223c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:44.496931 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.496838    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/641bf342-016f-4d7e-a14c-1f5abdafceb3-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4\" (UID: \"641bf342-016f-4d7e-a14c-1f5abdafceb3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4"
Sep 16 12:11:44.496931 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.496887    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc22k\" (UniqueName: \"kubernetes.io/projected/641bf342-016f-4d7e-a14c-1f5abdafceb3-kube-api-access-xc22k\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4\" (UID: \"641bf342-016f-4d7e-a14c-1f5abdafceb3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4"
Sep 16 12:11:44.496931 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.496912    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/641bf342-016f-4d7e-a14c-1f5abdafceb3-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4\" (UID: \"641bf342-016f-4d7e-a14c-1f5abdafceb3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4"
Sep 16 12:11:44.497245 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.497227    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/641bf342-016f-4d7e-a14c-1f5abdafceb3-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4\" (UID: \"641bf342-016f-4d7e-a14c-1f5abdafceb3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4"
Sep 16 12:11:44.497286 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.497253    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/641bf342-016f-4d7e-a14c-1f5abdafceb3-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4\" (UID: \"641bf342-016f-4d7e-a14c-1f5abdafceb3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4"
Sep 16 12:11:44.505663 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.505628    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc22k\" (UniqueName: \"kubernetes.io/projected/641bf342-016f-4d7e-a14c-1f5abdafceb3-kube-api-access-xc22k\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4\" (UID: \"641bf342-016f-4d7e-a14c-1f5abdafceb3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4"
Sep 16 12:11:44.511178 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.511148    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:44.672846 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.672811    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4"
Sep 16 12:11:44.788261 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.788192    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"]
Sep 16 12:11:44.790455 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:11:44.790429    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eed7827_c019_4b9d_b655_9c1c36223c1d.slice/crio-6c2f397e5dad29466cda5f6736c546336edac60463d4c7c35ae5cfc00b80e832 WatchSource:0}: Error finding container 6c2f397e5dad29466cda5f6736c546336edac60463d4c7c35ae5cfc00b80e832: Status 404 returned error can't find the container with id 6c2f397e5dad29466cda5f6736c546336edac60463d4c7c35ae5cfc00b80e832
Sep 16 12:11:44.945454 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:44.945429    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4"]
Sep 16 12:11:44.947257 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:11:44.947230    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641bf342_016f_4d7e_a14c_1f5abdafceb3.slice/crio-8b6b98a424d137c797e5b6f9119dfc3d08c0dc6901f0c7bc17a1993432b4879d WatchSource:0}: Error finding container 8b6b98a424d137c797e5b6f9119dfc3d08c0dc6901f0c7bc17a1993432b4879d: Status 404 returned error can't find the container with id 8b6b98a424d137c797e5b6f9119dfc3d08c0dc6901f0c7bc17a1993432b4879d
Sep 16 12:11:45.596926 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:45.596873    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp" event={"ID":"5eed7827-c019-4b9d-b655-9c1c36223c1d","Type":"ContainerStarted","Data":"6c2f397e5dad29466cda5f6736c546336edac60463d4c7c35ae5cfc00b80e832"}
Sep 16 12:11:45.599554 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:45.599510    2582 generic.go:356] "Generic (PLEG): container finished" podID="641bf342-016f-4d7e-a14c-1f5abdafceb3" containerID="785ccfd921a23991d8d4654bb1e796ae5458f490d511904315cbf058d679100b" exitCode=0
Sep 16 12:11:45.599709 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:45.599597    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4" event={"ID":"641bf342-016f-4d7e-a14c-1f5abdafceb3","Type":"ContainerDied","Data":"785ccfd921a23991d8d4654bb1e796ae5458f490d511904315cbf058d679100b"}
Sep 16 12:11:45.599709 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:45.599675    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4" event={"ID":"641bf342-016f-4d7e-a14c-1f5abdafceb3","Type":"ContainerStarted","Data":"8b6b98a424d137c797e5b6f9119dfc3d08c0dc6901f0c7bc17a1993432b4879d"}
Sep 16 12:11:48.618795 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:48.618751    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp" event={"ID":"5eed7827-c019-4b9d-b655-9c1c36223c1d","Type":"ContainerStarted","Data":"4e499949f505f2c957de223cb5da012f084d6faf03f7b7e76f7792833b5b0cd1"}
Sep 16 12:11:48.620853 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:48.620825    2582 generic.go:356] "Generic (PLEG): container finished" podID="641bf342-016f-4d7e-a14c-1f5abdafceb3" containerID="8ebb609e9bab9e1db69cc6fe13ea7b919592f0881f8528b805b393053bb1e2d0" exitCode=0
Sep 16 12:11:48.620957 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:48.620909    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4" event={"ID":"641bf342-016f-4d7e-a14c-1f5abdafceb3","Type":"ContainerDied","Data":"8ebb609e9bab9e1db69cc6fe13ea7b919592f0881f8528b805b393053bb1e2d0"}
Sep 16 12:11:49.627394 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:49.627357    2582 generic.go:356] "Generic (PLEG): container finished" podID="641bf342-016f-4d7e-a14c-1f5abdafceb3" containerID="4e99bbc9a586d02578eae96ed4f34c7a82e06d832a9ceffefe5f38e6f6b497eb" exitCode=0
Sep 16 12:11:49.627888 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:49.627437    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4" event={"ID":"641bf342-016f-4d7e-a14c-1f5abdafceb3","Type":"ContainerDied","Data":"4e99bbc9a586d02578eae96ed4f34c7a82e06d832a9ceffefe5f38e6f6b497eb"}
Sep 16 12:11:49.629168 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:49.629143    2582 generic.go:356] "Generic (PLEG): container finished" podID="5eed7827-c019-4b9d-b655-9c1c36223c1d" containerID="4e499949f505f2c957de223cb5da012f084d6faf03f7b7e76f7792833b5b0cd1" exitCode=0
Sep 16 12:11:49.629279 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:49.629189    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp" event={"ID":"5eed7827-c019-4b9d-b655-9c1c36223c1d","Type":"ContainerDied","Data":"4e499949f505f2c957de223cb5da012f084d6faf03f7b7e76f7792833b5b0cd1"}
Sep 16 12:11:50.635530 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:50.635492    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp" event={"ID":"5eed7827-c019-4b9d-b655-9c1c36223c1d","Type":"ContainerStarted","Data":"5ada8c9117dd17440b1d56d4626a8d0928ad2f7a3d209becd5df483e2964ca3e"}
Sep 16 12:11:50.635967 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:50.635782    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:11:50.659601 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:50.659556    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp" podStartSLOduration=2.051650004 podStartE2EDuration="6.659539256s" totalImagesPullingTime="3.438486471s" totalInitContainerRuntime="1.169402781s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:11:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:11:50.658055444 +0000 UTC m=+669.908592994" watchObservedRunningTime="2026-09-16 12:11:50.659539256 +0000 UTC m=+669.910076805"
Sep 16 12:11:50.770803 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:50.770781    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4"
Sep 16 12:11:50.858360 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:50.858321    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/641bf342-016f-4d7e-a14c-1f5abdafceb3-util\") pod \"641bf342-016f-4d7e-a14c-1f5abdafceb3\" (UID: \"641bf342-016f-4d7e-a14c-1f5abdafceb3\") "
Sep 16 12:11:50.858570 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:50.858411    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xc22k\" (UniqueName: \"kubernetes.io/projected/641bf342-016f-4d7e-a14c-1f5abdafceb3-kube-api-access-xc22k\") pod \"641bf342-016f-4d7e-a14c-1f5abdafceb3\" (UID: \"641bf342-016f-4d7e-a14c-1f5abdafceb3\") "
Sep 16 12:11:50.858570 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:50.858478    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/641bf342-016f-4d7e-a14c-1f5abdafceb3-bundle\") pod \"641bf342-016f-4d7e-a14c-1f5abdafceb3\" (UID: \"641bf342-016f-4d7e-a14c-1f5abdafceb3\") "
Sep 16 12:11:50.859179 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:50.859151    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641bf342-016f-4d7e-a14c-1f5abdafceb3-bundle" (OuterVolumeSpecName: "bundle") pod "641bf342-016f-4d7e-a14c-1f5abdafceb3" (UID: "641bf342-016f-4d7e-a14c-1f5abdafceb3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:11:50.860700 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:50.860669    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641bf342-016f-4d7e-a14c-1f5abdafceb3-kube-api-access-xc22k" (OuterVolumeSpecName: "kube-api-access-xc22k") pod "641bf342-016f-4d7e-a14c-1f5abdafceb3" (UID: "641bf342-016f-4d7e-a14c-1f5abdafceb3"). InnerVolumeSpecName "kube-api-access-xc22k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:11:50.864502 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:50.864478    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641bf342-016f-4d7e-a14c-1f5abdafceb3-util" (OuterVolumeSpecName: "util") pod "641bf342-016f-4d7e-a14c-1f5abdafceb3" (UID: "641bf342-016f-4d7e-a14c-1f5abdafceb3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:11:50.959269 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:50.959186    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc22k\" (UniqueName: \"kubernetes.io/projected/641bf342-016f-4d7e-a14c-1f5abdafceb3-kube-api-access-xc22k\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:11:50.959269 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:50.959216    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/641bf342-016f-4d7e-a14c-1f5abdafceb3-bundle\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:11:50.959269 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:50.959224    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/641bf342-016f-4d7e-a14c-1f5abdafceb3-util\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:11:51.641606 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:51.641569    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4" event={"ID":"641bf342-016f-4d7e-a14c-1f5abdafceb3","Type":"ContainerDied","Data":"8b6b98a424d137c797e5b6f9119dfc3d08c0dc6901f0c7bc17a1993432b4879d"}
Sep 16 12:11:51.641606 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:51.641599    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gdpl4"
Sep 16 12:11:51.642123 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:11:51.641640    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b6b98a424d137c797e5b6f9119dfc3d08c0dc6901f0c7bc17a1993432b4879d"
Sep 16 12:12:01.644324 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:01.644295    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7887d7f7b8-6v6jp"
Sep 16 12:12:05.015361 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.015327    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4"]
Sep 16 12:12:05.015759 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.015635    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="641bf342-016f-4d7e-a14c-1f5abdafceb3" containerName="pull"
Sep 16 12:12:05.015759 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.015647    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="641bf342-016f-4d7e-a14c-1f5abdafceb3" containerName="pull"
Sep 16 12:12:05.015759 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.015660    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="641bf342-016f-4d7e-a14c-1f5abdafceb3" containerName="extract"
Sep 16 12:12:05.015759 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.015667    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="641bf342-016f-4d7e-a14c-1f5abdafceb3" containerName="extract"
Sep 16 12:12:05.015759 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.015674    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="641bf342-016f-4d7e-a14c-1f5abdafceb3" containerName="util"
Sep 16 12:12:05.015759 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.015680    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="641bf342-016f-4d7e-a14c-1f5abdafceb3" containerName="util"
Sep 16 12:12:05.015960 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.015770    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="641bf342-016f-4d7e-a14c-1f5abdafceb3" containerName="extract"
Sep 16 12:12:05.018717 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.018696    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4"
Sep 16 12:12:05.022759 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.022724    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 12:12:05.022874 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.022804    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gdwph\""
Sep 16 12:12:05.023107 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.023093    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 12:12:05.037244 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.037217    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4"]
Sep 16 12:12:05.085507 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.085475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630a5bd9-f0fc-4806-9cba-aa16fd039c86-tls-certs\") pod \"kube-auth-proxy-55f6c77cbf-64gz4\" (UID: \"630a5bd9-f0fc-4806-9cba-aa16fd039c86\") " pod="openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4"
Sep 16 12:12:05.085696 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.085515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/630a5bd9-f0fc-4806-9cba-aa16fd039c86-tmp\") pod \"kube-auth-proxy-55f6c77cbf-64gz4\" (UID: \"630a5bd9-f0fc-4806-9cba-aa16fd039c86\") " pod="openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4"
Sep 16 12:12:05.085696 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.085590    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgscj\" (UniqueName: \"kubernetes.io/projected/630a5bd9-f0fc-4806-9cba-aa16fd039c86-kube-api-access-cgscj\") pod \"kube-auth-proxy-55f6c77cbf-64gz4\" (UID: \"630a5bd9-f0fc-4806-9cba-aa16fd039c86\") " pod="openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4"
Sep 16 12:12:05.186989 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.186949    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630a5bd9-f0fc-4806-9cba-aa16fd039c86-tls-certs\") pod \"kube-auth-proxy-55f6c77cbf-64gz4\" (UID: \"630a5bd9-f0fc-4806-9cba-aa16fd039c86\") " pod="openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4"
Sep 16 12:12:05.186989 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.186993    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/630a5bd9-f0fc-4806-9cba-aa16fd039c86-tmp\") pod \"kube-auth-proxy-55f6c77cbf-64gz4\" (UID: \"630a5bd9-f0fc-4806-9cba-aa16fd039c86\") " pod="openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4"
Sep 16 12:12:05.187264 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.187036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgscj\" (UniqueName: \"kubernetes.io/projected/630a5bd9-f0fc-4806-9cba-aa16fd039c86-kube-api-access-cgscj\") pod \"kube-auth-proxy-55f6c77cbf-64gz4\" (UID: \"630a5bd9-f0fc-4806-9cba-aa16fd039c86\") " pod="openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4"
Sep 16 12:12:05.187264 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:12:05.187116    2582 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 16 12:12:05.187264 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:12:05.187210    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/630a5bd9-f0fc-4806-9cba-aa16fd039c86-tls-certs podName:630a5bd9-f0fc-4806-9cba-aa16fd039c86 nodeName:}" failed. No retries permitted until 2026-09-16 12:12:05.687189785 +0000 UTC m=+684.937727315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/630a5bd9-f0fc-4806-9cba-aa16fd039c86-tls-certs") pod "kube-auth-proxy-55f6c77cbf-64gz4" (UID: "630a5bd9-f0fc-4806-9cba-aa16fd039c86") : secret "kube-auth-proxy-tls" not found
Sep 16 12:12:05.189364 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.189347    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/630a5bd9-f0fc-4806-9cba-aa16fd039c86-tmp\") pod \"kube-auth-proxy-55f6c77cbf-64gz4\" (UID: \"630a5bd9-f0fc-4806-9cba-aa16fd039c86\") " pod="openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4"
Sep 16 12:12:05.197441 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.197414    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgscj\" (UniqueName: \"kubernetes.io/projected/630a5bd9-f0fc-4806-9cba-aa16fd039c86-kube-api-access-cgscj\") pod \"kube-auth-proxy-55f6c77cbf-64gz4\" (UID: \"630a5bd9-f0fc-4806-9cba-aa16fd039c86\") " pod="openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4"
Sep 16 12:12:05.691819 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.691771    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630a5bd9-f0fc-4806-9cba-aa16fd039c86-tls-certs\") pod \"kube-auth-proxy-55f6c77cbf-64gz4\" (UID: \"630a5bd9-f0fc-4806-9cba-aa16fd039c86\") " pod="openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4"
Sep 16 12:12:05.694376 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.694356    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630a5bd9-f0fc-4806-9cba-aa16fd039c86-tls-certs\") pod \"kube-auth-proxy-55f6c77cbf-64gz4\" (UID: \"630a5bd9-f0fc-4806-9cba-aa16fd039c86\") " pod="openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4"
Sep 16 12:12:05.928010 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:05.927972    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4"
Sep 16 12:12:06.211523 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:06.211496    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4"]
Sep 16 12:12:06.214024 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:12:06.213993    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630a5bd9_f0fc_4806_9cba_aa16fd039c86.slice/crio-b879b14a7d35595abe6ce6ea5128b7bcd8d46f1a4ca9580e8eff9c20b97ed0bb WatchSource:0}: Error finding container b879b14a7d35595abe6ce6ea5128b7bcd8d46f1a4ca9580e8eff9c20b97ed0bb: Status 404 returned error can't find the container with id b879b14a7d35595abe6ce6ea5128b7bcd8d46f1a4ca9580e8eff9c20b97ed0bb
Sep 16 12:12:06.710058 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:06.710022    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4" event={"ID":"630a5bd9-f0fc-4806-9cba-aa16fd039c86","Type":"ContainerStarted","Data":"b879b14a7d35595abe6ce6ea5128b7bcd8d46f1a4ca9580e8eff9c20b97ed0bb"}
Sep 16 12:12:09.089725 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.089686    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74f74598d6-pzbjz"]
Sep 16 12:12:09.093117 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.093091    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.104671 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.104642    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74f74598d6-pzbjz"]
Sep 16 12:12:09.225779 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.225729    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75583f57-60b9-4d0a-9795-de7792fea195-console-serving-cert\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.225966 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.225810    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75583f57-60b9-4d0a-9795-de7792fea195-service-ca\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.225966 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.225833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nszjj\" (UniqueName: \"kubernetes.io/projected/75583f57-60b9-4d0a-9795-de7792fea195-kube-api-access-nszjj\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.225966 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.225857    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75583f57-60b9-4d0a-9795-de7792fea195-oauth-serving-cert\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.225966 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.225884    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75583f57-60b9-4d0a-9795-de7792fea195-console-config\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.226222 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.225977    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75583f57-60b9-4d0a-9795-de7792fea195-trusted-ca-bundle\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.226222 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.226016    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75583f57-60b9-4d0a-9795-de7792fea195-console-oauth-config\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.326810 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.326769    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75583f57-60b9-4d0a-9795-de7792fea195-service-ca\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.326810 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.326814    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nszjj\" (UniqueName: \"kubernetes.io/projected/75583f57-60b9-4d0a-9795-de7792fea195-kube-api-access-nszjj\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.327080 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.326835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75583f57-60b9-4d0a-9795-de7792fea195-oauth-serving-cert\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.327080 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.326860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75583f57-60b9-4d0a-9795-de7792fea195-console-config\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.327080 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.326887    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75583f57-60b9-4d0a-9795-de7792fea195-trusted-ca-bundle\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.327080 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.326908    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75583f57-60b9-4d0a-9795-de7792fea195-console-oauth-config\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.327080 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.326939    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75583f57-60b9-4d0a-9795-de7792fea195-console-serving-cert\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.327659 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.327634    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75583f57-60b9-4d0a-9795-de7792fea195-console-config\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.327809 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.327769    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75583f57-60b9-4d0a-9795-de7792fea195-service-ca\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.327913 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.327805    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75583f57-60b9-4d0a-9795-de7792fea195-oauth-serving-cert\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.327957 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.327900    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75583f57-60b9-4d0a-9795-de7792fea195-trusted-ca-bundle\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.329577 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.329558    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75583f57-60b9-4d0a-9795-de7792fea195-console-serving-cert\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.329691 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.329673    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75583f57-60b9-4d0a-9795-de7792fea195-console-oauth-config\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.334865 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.334844    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nszjj\" (UniqueName: \"kubernetes.io/projected/75583f57-60b9-4d0a-9795-de7792fea195-kube-api-access-nszjj\") pod \"console-74f74598d6-pzbjz\" (UID: \"75583f57-60b9-4d0a-9795-de7792fea195\") " pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.404222 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.404141    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:09.679416 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.679367    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74f74598d6-pzbjz"]
Sep 16 12:12:09.681922 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:12:09.681890    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75583f57_60b9_4d0a_9795_de7792fea195.slice/crio-7b40d994cdb8f4d4fad31b7297b2e76a6d18afef85ce440cb23f9c99166959bc WatchSource:0}: Error finding container 7b40d994cdb8f4d4fad31b7297b2e76a6d18afef85ce440cb23f9c99166959bc: Status 404 returned error can't find the container with id 7b40d994cdb8f4d4fad31b7297b2e76a6d18afef85ce440cb23f9c99166959bc
Sep 16 12:12:09.734488 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.734451    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f74598d6-pzbjz" event={"ID":"75583f57-60b9-4d0a-9795-de7792fea195","Type":"ContainerStarted","Data":"7b40d994cdb8f4d4fad31b7297b2e76a6d18afef85ce440cb23f9c99166959bc"}
Sep 16 12:12:09.736184 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.736123    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4" event={"ID":"630a5bd9-f0fc-4806-9cba-aa16fd039c86","Type":"ContainerStarted","Data":"694cffe52aa2071de2363fc22d1a7e31cf8f1ebd74f248223c87ea514f7e9b5a"}
Sep 16 12:12:09.752452 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:09.752408    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-55f6c77cbf-64gz4" podStartSLOduration=2.820545983 podStartE2EDuration="5.752388599s" totalImagesPullingTime="2.931842616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:09.750374941 +0000 UTC m=+689.000912491" watchObservedRunningTime="2026-09-16 12:12:09.752388599 +0000 UTC m=+689.002926148"
Sep 16 12:12:10.742799 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:10.742764    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f74598d6-pzbjz" event={"ID":"75583f57-60b9-4d0a-9795-de7792fea195","Type":"ContainerStarted","Data":"edc8b94915602b715156bdc68e8b2626478f71ebbce2147e4bbe12d943fae324"}
Sep 16 12:12:10.761930 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:10.761864    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74f74598d6-pzbjz" podStartSLOduration=1.761848054 podStartE2EDuration="1.761848054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:10.760674863 +0000 UTC m=+690.011212439" watchObservedRunningTime="2026-09-16 12:12:10.761848054 +0000 UTC m=+690.012385604"
Sep 16 12:12:19.404404 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:19.404371    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:19.404888 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:19.404427    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:19.409325 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:19.409298    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:19.789308 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:19.789234    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74f74598d6-pzbjz"
Sep 16 12:12:19.832974 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:19.832932    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75f77c54b-lr9h6"]
Sep 16 12:12:20.532668 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.532630    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"]
Sep 16 12:12:20.537172 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.537156    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"
Sep 16 12:12:20.539944 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.539923    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:12:20.540070 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.539944    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-sw8ht\""
Sep 16 12:12:20.541437 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.541418    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 12:12:20.541551 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.541437    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 12:12:20.541551 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.541499    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 12:12:20.541642 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.541551    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 12:12:20.545323 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.545304    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"]
Sep 16 12:12:20.626158 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.626119    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmgjx\" (UniqueName: \"kubernetes.io/projected/93e1607f-5197-4a64-ae99-9de6a620763e-kube-api-access-rmgjx\") pod \"lws-controller-manager-c7f7d6d7f-5dvlv\" (UID: \"93e1607f-5197-4a64-ae99-9de6a620763e\") " pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"
Sep 16 12:12:20.626362 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.626182    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93e1607f-5197-4a64-ae99-9de6a620763e-cert\") pod \"lws-controller-manager-c7f7d6d7f-5dvlv\" (UID: \"93e1607f-5197-4a64-ae99-9de6a620763e\") " pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"
Sep 16 12:12:20.626362 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.626212    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/93e1607f-5197-4a64-ae99-9de6a620763e-metrics-cert\") pod \"lws-controller-manager-c7f7d6d7f-5dvlv\" (UID: \"93e1607f-5197-4a64-ae99-9de6a620763e\") " pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"
Sep 16 12:12:20.626362 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.626231    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/93e1607f-5197-4a64-ae99-9de6a620763e-manager-config\") pod \"lws-controller-manager-c7f7d6d7f-5dvlv\" (UID: \"93e1607f-5197-4a64-ae99-9de6a620763e\") " pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"
Sep 16 12:12:20.727030 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.726992    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93e1607f-5197-4a64-ae99-9de6a620763e-cert\") pod \"lws-controller-manager-c7f7d6d7f-5dvlv\" (UID: \"93e1607f-5197-4a64-ae99-9de6a620763e\") " pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"
Sep 16 12:12:20.727238 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.727040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/93e1607f-5197-4a64-ae99-9de6a620763e-metrics-cert\") pod \"lws-controller-manager-c7f7d6d7f-5dvlv\" (UID: \"93e1607f-5197-4a64-ae99-9de6a620763e\") " pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"
Sep 16 12:12:20.727238 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.727060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/93e1607f-5197-4a64-ae99-9de6a620763e-manager-config\") pod \"lws-controller-manager-c7f7d6d7f-5dvlv\" (UID: \"93e1607f-5197-4a64-ae99-9de6a620763e\") " pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"
Sep 16 12:12:20.727238 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.727111    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmgjx\" (UniqueName: \"kubernetes.io/projected/93e1607f-5197-4a64-ae99-9de6a620763e-kube-api-access-rmgjx\") pod \"lws-controller-manager-c7f7d6d7f-5dvlv\" (UID: \"93e1607f-5197-4a64-ae99-9de6a620763e\") " pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"
Sep 16 12:12:20.727907 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.727808    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/93e1607f-5197-4a64-ae99-9de6a620763e-manager-config\") pod \"lws-controller-manager-c7f7d6d7f-5dvlv\" (UID: \"93e1607f-5197-4a64-ae99-9de6a620763e\") " pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"
Sep 16 12:12:20.729834 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.729809    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93e1607f-5197-4a64-ae99-9de6a620763e-cert\") pod \"lws-controller-manager-c7f7d6d7f-5dvlv\" (UID: \"93e1607f-5197-4a64-ae99-9de6a620763e\") " pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"
Sep 16 12:12:20.729918 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.729836    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/93e1607f-5197-4a64-ae99-9de6a620763e-metrics-cert\") pod \"lws-controller-manager-c7f7d6d7f-5dvlv\" (UID: \"93e1607f-5197-4a64-ae99-9de6a620763e\") " pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"
Sep 16 12:12:20.735701 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.735677    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmgjx\" (UniqueName: \"kubernetes.io/projected/93e1607f-5197-4a64-ae99-9de6a620763e-kube-api-access-rmgjx\") pod \"lws-controller-manager-c7f7d6d7f-5dvlv\" (UID: \"93e1607f-5197-4a64-ae99-9de6a620763e\") " pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"
Sep 16 12:12:20.847602 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:20.847567    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"
Sep 16 12:12:21.123499 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:21.123475    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"]
Sep 16 12:12:21.125642 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:12:21.125609    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e1607f_5197_4a64_ae99_9de6a620763e.slice/crio-2239fc39b60638ffc6478a063952ea858e8c285a3cd5fce14eb14fde1fc1ae5d WatchSource:0}: Error finding container 2239fc39b60638ffc6478a063952ea858e8c285a3cd5fce14eb14fde1fc1ae5d: Status 404 returned error can't find the container with id 2239fc39b60638ffc6478a063952ea858e8c285a3cd5fce14eb14fde1fc1ae5d
Sep 16 12:12:21.797686 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:21.797642    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv" event={"ID":"93e1607f-5197-4a64-ae99-9de6a620763e","Type":"ContainerStarted","Data":"2239fc39b60638ffc6478a063952ea858e8c285a3cd5fce14eb14fde1fc1ae5d"}
Sep 16 12:12:22.804622 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:22.804583    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv" event={"ID":"93e1607f-5197-4a64-ae99-9de6a620763e","Type":"ContainerStarted","Data":"94abc993cc76c42942ce2042d6c0a5cfe7734a8bf940d1bd24550b876f554c9d"}
Sep 16 12:12:22.805126 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:22.804758    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"
Sep 16 12:12:22.833495 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:22.833430    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv" podStartSLOduration=1.3525191030000001 podStartE2EDuration="2.833412967s" totalImagesPullingTime="1.480893864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:22.829162829 +0000 UTC m=+702.079700378" watchObservedRunningTime="2026-09-16 12:12:22.833412967 +0000 UTC m=+702.083950584"
Sep 16 12:12:33.811775 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:33.811664    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-c7f7d6d7f-5dvlv"
Sep 16 12:12:44.853278 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:44.853231    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75f77c54b-lr9h6" podUID="2c8026c6-c14f-42fb-8664-c525009a114f" containerName="console" containerID="cri-o://12672d869742543eea4c94cea579b08e72d32e9e258cd01a34d512926fb26266" gracePeriod=15
Sep 16 12:12:45.090604 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.090579    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:12:45.114433 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.114349    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c8026c6-c14f-42fb-8664-c525009a114f-console-oauth-config\") pod \"2c8026c6-c14f-42fb-8664-c525009a114f\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") "
Sep 16 12:12:45.114433 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.114409    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-oauth-serving-cert\") pod \"2c8026c6-c14f-42fb-8664-c525009a114f\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") "
Sep 16 12:12:45.114655 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.114455    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c8026c6-c14f-42fb-8664-c525009a114f-console-serving-cert\") pod \"2c8026c6-c14f-42fb-8664-c525009a114f\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") "
Sep 16 12:12:45.114655 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.114477    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-service-ca\") pod \"2c8026c6-c14f-42fb-8664-c525009a114f\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") "
Sep 16 12:12:45.114655 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.114509    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-trusted-ca-bundle\") pod \"2c8026c6-c14f-42fb-8664-c525009a114f\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") "
Sep 16 12:12:45.114655 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.114565    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wstxt\" (UniqueName: \"kubernetes.io/projected/2c8026c6-c14f-42fb-8664-c525009a114f-kube-api-access-wstxt\") pod \"2c8026c6-c14f-42fb-8664-c525009a114f\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") "
Sep 16 12:12:45.114909 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.114831    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2c8026c6-c14f-42fb-8664-c525009a114f" (UID: "2c8026c6-c14f-42fb-8664-c525009a114f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:12:45.114909 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.114874    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-service-ca" (OuterVolumeSpecName: "service-ca") pod "2c8026c6-c14f-42fb-8664-c525009a114f" (UID: "2c8026c6-c14f-42fb-8664-c525009a114f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:12:45.114909 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.114890    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-console-config\") pod \"2c8026c6-c14f-42fb-8664-c525009a114f\" (UID: \"2c8026c6-c14f-42fb-8664-c525009a114f\") "
Sep 16 12:12:45.115288 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.115131    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2c8026c6-c14f-42fb-8664-c525009a114f" (UID: "2c8026c6-c14f-42fb-8664-c525009a114f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:12:45.115288 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.115276    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-oauth-serving-cert\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:12:45.115288 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.115278    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-console-config" (OuterVolumeSpecName: "console-config") pod "2c8026c6-c14f-42fb-8664-c525009a114f" (UID: "2c8026c6-c14f-42fb-8664-c525009a114f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:12:45.115288 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.115298    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:12:45.116914 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.116884    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8026c6-c14f-42fb-8664-c525009a114f-kube-api-access-wstxt" (OuterVolumeSpecName: "kube-api-access-wstxt") pod "2c8026c6-c14f-42fb-8664-c525009a114f" (UID: "2c8026c6-c14f-42fb-8664-c525009a114f"). InnerVolumeSpecName "kube-api-access-wstxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:12:45.117349 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.117327    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8026c6-c14f-42fb-8664-c525009a114f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2c8026c6-c14f-42fb-8664-c525009a114f" (UID: "2c8026c6-c14f-42fb-8664-c525009a114f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:12:45.117349 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.117342    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8026c6-c14f-42fb-8664-c525009a114f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2c8026c6-c14f-42fb-8664-c525009a114f" (UID: "2c8026c6-c14f-42fb-8664-c525009a114f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:12:45.216037 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.215999    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wstxt\" (UniqueName: \"kubernetes.io/projected/2c8026c6-c14f-42fb-8664-c525009a114f-kube-api-access-wstxt\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:12:45.216037 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.216037    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-console-config\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:12:45.216251 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.216053    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c8026c6-c14f-42fb-8664-c525009a114f-console-oauth-config\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:12:45.216251 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.216070    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c8026c6-c14f-42fb-8664-c525009a114f-console-serving-cert\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:12:45.216251 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.216083    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c8026c6-c14f-42fb-8664-c525009a114f-trusted-ca-bundle\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:12:45.918670 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.918636    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c8026c6-c14f-42fb-8664-c525009a114f" containerID="12672d869742543eea4c94cea579b08e72d32e9e258cd01a34d512926fb26266" exitCode=0
Sep 16 12:12:45.919183 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.918682    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f77c54b-lr9h6"
Sep 16 12:12:45.919183 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.918722    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f77c54b-lr9h6" event={"ID":"2c8026c6-c14f-42fb-8664-c525009a114f","Type":"ContainerDied","Data":"12672d869742543eea4c94cea579b08e72d32e9e258cd01a34d512926fb26266"}
Sep 16 12:12:45.919183 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.918802    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f77c54b-lr9h6" event={"ID":"2c8026c6-c14f-42fb-8664-c525009a114f","Type":"ContainerDied","Data":"dc8955d9eb0ea499bbaf6727474f26107493d549bcfa6885dffb59cf7b3d14a9"}
Sep 16 12:12:45.919183 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.918831    2582 scope.go:117] "RemoveContainer" containerID="12672d869742543eea4c94cea579b08e72d32e9e258cd01a34d512926fb26266"
Sep 16 12:12:45.927900 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.927878    2582 scope.go:117] "RemoveContainer" containerID="12672d869742543eea4c94cea579b08e72d32e9e258cd01a34d512926fb26266"
Sep 16 12:12:45.928166 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:12:45.928147    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12672d869742543eea4c94cea579b08e72d32e9e258cd01a34d512926fb26266\": container with ID starting with 12672d869742543eea4c94cea579b08e72d32e9e258cd01a34d512926fb26266 not found: ID does not exist" containerID="12672d869742543eea4c94cea579b08e72d32e9e258cd01a34d512926fb26266"
Sep 16 12:12:45.928211 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.928176    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12672d869742543eea4c94cea579b08e72d32e9e258cd01a34d512926fb26266"} err="failed to get container status \"12672d869742543eea4c94cea579b08e72d32e9e258cd01a34d512926fb26266\": rpc error: code = NotFound desc = could not find container \"12672d869742543eea4c94cea579b08e72d32e9e258cd01a34d512926fb26266\": container with ID starting with 12672d869742543eea4c94cea579b08e72d32e9e258cd01a34d512926fb26266 not found: ID does not exist"
Sep 16 12:12:45.935268 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.935244    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75f77c54b-lr9h6"]
Sep 16 12:12:45.938912 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:45.938894    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75f77c54b-lr9h6"]
Sep 16 12:12:47.211203 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:12:47.211168    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c8026c6-c14f-42fb-8664-c525009a114f" path="/var/lib/kubelet/pods/2c8026c6-c14f-42fb-8664-c525009a114f/volumes"
Sep 16 12:13:20.503016 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.502978    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9"]
Sep 16 12:13:20.503443 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.503310    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c8026c6-c14f-42fb-8664-c525009a114f" containerName="console"
Sep 16 12:13:20.503443 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.503322    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c8026c6-c14f-42fb-8664-c525009a114f" containerName="console"
Sep 16 12:13:20.503443 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.503379    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c8026c6-c14f-42fb-8664-c525009a114f" containerName="console"
Sep 16 12:13:20.505939 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.505922    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9"
Sep 16 12:13:20.508639 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.508614    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:13:20.508767 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.508614    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:13:20.509793 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.509772    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bp9w9\""
Sep 16 12:13:20.515329 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.515303    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9"]
Sep 16 12:13:20.603385 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.603349    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj"]
Sep 16 12:13:20.605961 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.605939    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj"
Sep 16 12:13:20.614919 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.614880    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc5cae68-af62-4c78-84e3-52d22ee21b53-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9\" (UID: \"bc5cae68-af62-4c78-84e3-52d22ee21b53\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9"
Sep 16 12:13:20.615074 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.614975    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s99pp\" (UniqueName: \"kubernetes.io/projected/bc5cae68-af62-4c78-84e3-52d22ee21b53-kube-api-access-s99pp\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9\" (UID: \"bc5cae68-af62-4c78-84e3-52d22ee21b53\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9"
Sep 16 12:13:20.615074 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.615017    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj"]
Sep 16 12:13:20.615074 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.615033    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc5cae68-af62-4c78-84e3-52d22ee21b53-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9\" (UID: \"bc5cae68-af62-4c78-84e3-52d22ee21b53\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9"
Sep 16 12:13:20.703795 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.703757    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c"]
Sep 16 12:13:20.706159 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.706140    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c"
Sep 16 12:13:20.715961 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.715933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s99pp\" (UniqueName: \"kubernetes.io/projected/bc5cae68-af62-4c78-84e3-52d22ee21b53-kube-api-access-s99pp\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9\" (UID: \"bc5cae68-af62-4c78-84e3-52d22ee21b53\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9"
Sep 16 12:13:20.716120 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.715993    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc5cae68-af62-4c78-84e3-52d22ee21b53-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9\" (UID: \"bc5cae68-af62-4c78-84e3-52d22ee21b53\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9"
Sep 16 12:13:20.716120 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.716040    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b1122d0-9c42-4878-ba89-6fee21c3dac5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj\" (UID: \"1b1122d0-9c42-4878-ba89-6fee21c3dac5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj"
Sep 16 12:13:20.716120 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.716067    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc5cae68-af62-4c78-84e3-52d22ee21b53-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9\" (UID: \"bc5cae68-af62-4c78-84e3-52d22ee21b53\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9"
Sep 16 12:13:20.716287 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.716223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2hg9\" (UniqueName: \"kubernetes.io/projected/1b1122d0-9c42-4878-ba89-6fee21c3dac5-kube-api-access-w2hg9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj\" (UID: \"1b1122d0-9c42-4878-ba89-6fee21c3dac5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj"
Sep 16 12:13:20.716347 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.716285    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b1122d0-9c42-4878-ba89-6fee21c3dac5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj\" (UID: \"1b1122d0-9c42-4878-ba89-6fee21c3dac5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj"
Sep 16 12:13:20.716543 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.716517    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc5cae68-af62-4c78-84e3-52d22ee21b53-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9\" (UID: \"bc5cae68-af62-4c78-84e3-52d22ee21b53\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9"
Sep 16 12:13:20.716668 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.716577    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc5cae68-af62-4c78-84e3-52d22ee21b53-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9\" (UID: \"bc5cae68-af62-4c78-84e3-52d22ee21b53\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9"
Sep 16 12:13:20.716777 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.716762    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c"]
Sep 16 12:13:20.725038 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.725015    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s99pp\" (UniqueName: \"kubernetes.io/projected/bc5cae68-af62-4c78-84e3-52d22ee21b53-kube-api-access-s99pp\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9\" (UID: \"bc5cae68-af62-4c78-84e3-52d22ee21b53\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9"
Sep 16 12:13:20.802681 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.802595    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5"]
Sep 16 12:13:20.805192 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.805174    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5"
Sep 16 12:13:20.814578 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.814550    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5"]
Sep 16 12:13:20.815639 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.815622    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9"
Sep 16 12:13:20.816980 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.816955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3103550-7364-4ee8-85b8-628b1bda8767-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c\" (UID: \"f3103550-7364-4ee8-85b8-628b1bda8767\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c"
Sep 16 12:13:20.817090 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.817000    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b1122d0-9c42-4878-ba89-6fee21c3dac5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj\" (UID: \"1b1122d0-9c42-4878-ba89-6fee21c3dac5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj"
Sep 16 12:13:20.817090 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.817024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2hg9\" (UniqueName: \"kubernetes.io/projected/1b1122d0-9c42-4878-ba89-6fee21c3dac5-kube-api-access-w2hg9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj\" (UID: \"1b1122d0-9c42-4878-ba89-6fee21c3dac5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj"
Sep 16 12:13:20.817090 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.817051    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3103550-7364-4ee8-85b8-628b1bda8767-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c\" (UID: \"f3103550-7364-4ee8-85b8-628b1bda8767\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c"
Sep 16 12:13:20.817090 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.817082    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b1122d0-9c42-4878-ba89-6fee21c3dac5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj\" (UID: \"1b1122d0-9c42-4878-ba89-6fee21c3dac5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj"
Sep 16 12:13:20.817301 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.817166    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjz9q\" (UniqueName: \"kubernetes.io/projected/f3103550-7364-4ee8-85b8-628b1bda8767-kube-api-access-xjz9q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c\" (UID: \"f3103550-7364-4ee8-85b8-628b1bda8767\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c"
Sep 16 12:13:20.817384 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.817363    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b1122d0-9c42-4878-ba89-6fee21c3dac5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj\" (UID: \"1b1122d0-9c42-4878-ba89-6fee21c3dac5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj"
Sep 16 12:13:20.817441 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.817399    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b1122d0-9c42-4878-ba89-6fee21c3dac5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj\" (UID: \"1b1122d0-9c42-4878-ba89-6fee21c3dac5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj"
Sep 16 12:13:20.828336 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.828303    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2hg9\" (UniqueName: \"kubernetes.io/projected/1b1122d0-9c42-4878-ba89-6fee21c3dac5-kube-api-access-w2hg9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj\" (UID: \"1b1122d0-9c42-4878-ba89-6fee21c3dac5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj"
Sep 16 12:13:20.916578 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.916545    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj"
Sep 16 12:13:20.917873 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.917842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3103550-7364-4ee8-85b8-628b1bda8767-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c\" (UID: \"f3103550-7364-4ee8-85b8-628b1bda8767\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c"
Sep 16 12:13:20.918009 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.917898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88278d5b-8511-4dbc-a32d-292a27f9c1bf-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5\" (UID: \"88278d5b-8511-4dbc-a32d-292a27f9c1bf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5"
Sep 16 12:13:20.918009 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.917942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjz9q\" (UniqueName: \"kubernetes.io/projected/f3103550-7364-4ee8-85b8-628b1bda8767-kube-api-access-xjz9q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c\" (UID: \"f3103550-7364-4ee8-85b8-628b1bda8767\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c"
Sep 16 12:13:20.918009 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.917981    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3103550-7364-4ee8-85b8-628b1bda8767-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c\" (UID: \"f3103550-7364-4ee8-85b8-628b1bda8767\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c"
Sep 16 12:13:20.918180 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.918009    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88278d5b-8511-4dbc-a32d-292a27f9c1bf-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5\" (UID: \"88278d5b-8511-4dbc-a32d-292a27f9c1bf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5"
Sep 16 12:13:20.918180 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.918052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd4ls\" (UniqueName: \"kubernetes.io/projected/88278d5b-8511-4dbc-a32d-292a27f9c1bf-kube-api-access-fd4ls\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5\" (UID: \"88278d5b-8511-4dbc-a32d-292a27f9c1bf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5"
Sep 16 12:13:20.918354 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.918333    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3103550-7364-4ee8-85b8-628b1bda8767-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c\" (UID: \"f3103550-7364-4ee8-85b8-628b1bda8767\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c"
Sep 16 12:13:20.918354 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.918341    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3103550-7364-4ee8-85b8-628b1bda8767-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c\" (UID: \"f3103550-7364-4ee8-85b8-628b1bda8767\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c"
Sep 16 12:13:20.926570 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:20.926541    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjz9q\" (UniqueName: \"kubernetes.io/projected/f3103550-7364-4ee8-85b8-628b1bda8767-kube-api-access-xjz9q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c\" (UID: \"f3103550-7364-4ee8-85b8-628b1bda8767\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c"
Sep 16 12:13:21.015509 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:21.015468    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c"
Sep 16 12:13:21.018623 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:21.018462    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd4ls\" (UniqueName: \"kubernetes.io/projected/88278d5b-8511-4dbc-a32d-292a27f9c1bf-kube-api-access-fd4ls\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5\" (UID: \"88278d5b-8511-4dbc-a32d-292a27f9c1bf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5"
Sep 16 12:13:21.018623 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:21.018515    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88278d5b-8511-4dbc-a32d-292a27f9c1bf-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5\" (UID: \"88278d5b-8511-4dbc-a32d-292a27f9c1bf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5"
Sep 16 12:13:21.018623 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:21.018568    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88278d5b-8511-4dbc-a32d-292a27f9c1bf-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5\" (UID: \"88278d5b-8511-4dbc-a32d-292a27f9c1bf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5"
Sep 16 12:13:21.018979 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:21.018961    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88278d5b-8511-4dbc-a32d-292a27f9c1bf-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5\" (UID: \"88278d5b-8511-4dbc-a32d-292a27f9c1bf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5"
Sep 16 12:13:21.019025 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:21.018980    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88278d5b-8511-4dbc-a32d-292a27f9c1bf-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5\" (UID: \"88278d5b-8511-4dbc-a32d-292a27f9c1bf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5"
Sep 16 12:13:21.027453 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:21.027433    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd4ls\" (UniqueName: \"kubernetes.io/projected/88278d5b-8511-4dbc-a32d-292a27f9c1bf-kube-api-access-fd4ls\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5\" (UID: \"88278d5b-8511-4dbc-a32d-292a27f9c1bf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5"
Sep 16 12:13:21.094630 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:21.094594    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9"]
Sep 16 12:13:21.095752 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:13:21.095712    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5cae68_af62_4c78_84e3_52d22ee21b53.slice/crio-7f2944f2353346e89e2da8f78fd2057b1205e082921657113da05d85a3f55aa3 WatchSource:0}: Error finding container 7f2944f2353346e89e2da8f78fd2057b1205e082921657113da05d85a3f55aa3: Status 404 returned error can't find the container with id 7f2944f2353346e89e2da8f78fd2057b1205e082921657113da05d85a3f55aa3
Sep 16 12:13:21.115048 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:21.115023    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5"
Sep 16 12:13:21.189948 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:21.189474    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj"]
Sep 16 12:13:21.199767 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:13:21.199694    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1122d0_9c42_4878_ba89_6fee21c3dac5.slice/crio-f84929a9e76bae54c4327751320d21475dc61e2dd974e99d1c4482a637d24eea WatchSource:0}: Error finding container f84929a9e76bae54c4327751320d21475dc61e2dd974e99d1c4482a637d24eea: Status 404 returned error can't find the container with id f84929a9e76bae54c4327751320d21475dc61e2dd974e99d1c4482a637d24eea
Sep 16 12:13:21.286336 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:21.286310    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c"]
Sep 16 12:13:21.287708 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:13:21.287682    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3103550_7364_4ee8_85b8_628b1bda8767.slice/crio-7da06f8b2684dc016661941ecbbd7f91e9bc57752db04cfecdd39ed0a9e2310d WatchSource:0}: Error finding container 7da06f8b2684dc016661941ecbbd7f91e9bc57752db04cfecdd39ed0a9e2310d: Status 404 returned error can't find the container with id 7da06f8b2684dc016661941ecbbd7f91e9bc57752db04cfecdd39ed0a9e2310d
Sep 16 12:13:21.387875 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:21.387851    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5"]
Sep 16 12:13:21.416629 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:13:21.416593    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88278d5b_8511_4dbc_a32d_292a27f9c1bf.slice/crio-af582349dafe3ea4ba308e8fcd98e8233ad50884ceaa37bf81d7ec2d89ff3673 WatchSource:0}: Error finding container af582349dafe3ea4ba308e8fcd98e8233ad50884ceaa37bf81d7ec2d89ff3673: Status 404 returned error can't find the container with id af582349dafe3ea4ba308e8fcd98e8233ad50884ceaa37bf81d7ec2d89ff3673
Sep 16 12:13:22.101134 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:22.101095    2582 generic.go:356] "Generic (PLEG): container finished" podID="bc5cae68-af62-4c78-84e3-52d22ee21b53" containerID="2dad0110f245512b725085a87f5babb0765194494fed820a847665e1f98021f9" exitCode=0
Sep 16 12:13:22.101682 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:22.101135    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9" event={"ID":"bc5cae68-af62-4c78-84e3-52d22ee21b53","Type":"ContainerDied","Data":"2dad0110f245512b725085a87f5babb0765194494fed820a847665e1f98021f9"}
Sep 16 12:13:22.101682 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:22.101193    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9" event={"ID":"bc5cae68-af62-4c78-84e3-52d22ee21b53","Type":"ContainerStarted","Data":"7f2944f2353346e89e2da8f78fd2057b1205e082921657113da05d85a3f55aa3"}
Sep 16 12:13:22.103026 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:22.103005    2582 generic.go:356] "Generic (PLEG): container finished" podID="88278d5b-8511-4dbc-a32d-292a27f9c1bf" containerID="9613aa136148e1cafa7bd476c8d454c85e46a94b6d85a898da7a2babb502f840" exitCode=0
Sep 16 12:13:22.103120 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:22.103094    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5" event={"ID":"88278d5b-8511-4dbc-a32d-292a27f9c1bf","Type":"ContainerDied","Data":"9613aa136148e1cafa7bd476c8d454c85e46a94b6d85a898da7a2babb502f840"}
Sep 16 12:13:22.103174 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:22.103140    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5" event={"ID":"88278d5b-8511-4dbc-a32d-292a27f9c1bf","Type":"ContainerStarted","Data":"af582349dafe3ea4ba308e8fcd98e8233ad50884ceaa37bf81d7ec2d89ff3673"}
Sep 16 12:13:22.104801 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:22.104777    2582 generic.go:356] "Generic (PLEG): container finished" podID="f3103550-7364-4ee8-85b8-628b1bda8767" containerID="73385186a1d95861ca6b5e14bac932c3dd8ef410af1e5676a6e9d5b60d4a39c9" exitCode=0
Sep 16 12:13:22.104914 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:22.104885    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c" event={"ID":"f3103550-7364-4ee8-85b8-628b1bda8767","Type":"ContainerDied","Data":"73385186a1d95861ca6b5e14bac932c3dd8ef410af1e5676a6e9d5b60d4a39c9"}
Sep 16 12:13:22.104976 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:22.104926    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c" event={"ID":"f3103550-7364-4ee8-85b8-628b1bda8767","Type":"ContainerStarted","Data":"7da06f8b2684dc016661941ecbbd7f91e9bc57752db04cfecdd39ed0a9e2310d"}
Sep 16 12:13:22.106697 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:22.106679    2582 generic.go:356] "Generic (PLEG): container finished" podID="1b1122d0-9c42-4878-ba89-6fee21c3dac5" containerID="bf870d11fc54fce881f5c5c20b2fffc4c6d1e4c4fab60333bffd3cb9d532a49a" exitCode=0
Sep 16 12:13:22.106806 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:22.106710    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj" event={"ID":"1b1122d0-9c42-4878-ba89-6fee21c3dac5","Type":"ContainerDied","Data":"bf870d11fc54fce881f5c5c20b2fffc4c6d1e4c4fab60333bffd3cb9d532a49a"}
Sep 16 12:13:22.106865 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:22.106810    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj" event={"ID":"1b1122d0-9c42-4878-ba89-6fee21c3dac5","Type":"ContainerStarted","Data":"f84929a9e76bae54c4327751320d21475dc61e2dd974e99d1c4482a637d24eea"}
Sep 16 12:13:23.114865 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:23.114832    2582 generic.go:356] "Generic (PLEG): container finished" podID="1b1122d0-9c42-4878-ba89-6fee21c3dac5" containerID="b45b547b65336dba13eb6cd27ea6bbd89ffcb79f5873f5175f20a88a4269e74e" exitCode=0
Sep 16 12:13:23.115222 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:23.114917    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj" event={"ID":"1b1122d0-9c42-4878-ba89-6fee21c3dac5","Type":"ContainerDied","Data":"b45b547b65336dba13eb6cd27ea6bbd89ffcb79f5873f5175f20a88a4269e74e"}
Sep 16 12:13:24.122432 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:24.122343    2582 generic.go:356] "Generic (PLEG): container finished" podID="88278d5b-8511-4dbc-a32d-292a27f9c1bf" containerID="78bcd9b9cc4f0c33287e20397204456e9fb9c07ed5daf295dfd64070ba51ea3e" exitCode=0
Sep 16 12:13:24.122862 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:24.122427    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5" event={"ID":"88278d5b-8511-4dbc-a32d-292a27f9c1bf","Type":"ContainerDied","Data":"78bcd9b9cc4f0c33287e20397204456e9fb9c07ed5daf295dfd64070ba51ea3e"}
Sep 16 12:13:24.124486 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:24.124465    2582 generic.go:356] "Generic (PLEG): container finished" podID="f3103550-7364-4ee8-85b8-628b1bda8767" containerID="22e32df8d316153302914c9f3e3d0675af3fb890eb23d211a4439e851f1de067" exitCode=0
Sep 16 12:13:24.124572 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:24.124546    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c" event={"ID":"f3103550-7364-4ee8-85b8-628b1bda8767","Type":"ContainerDied","Data":"22e32df8d316153302914c9f3e3d0675af3fb890eb23d211a4439e851f1de067"}
Sep 16 12:13:24.126997 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:24.126976    2582 generic.go:356] "Generic (PLEG): container finished" podID="1b1122d0-9c42-4878-ba89-6fee21c3dac5" containerID="880f5e01a6769d80fc910dfb8937f72994234c95da0636ce9a82d48d28698bdf" exitCode=0
Sep 16 12:13:24.127098 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:24.127054    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj" event={"ID":"1b1122d0-9c42-4878-ba89-6fee21c3dac5","Type":"ContainerDied","Data":"880f5e01a6769d80fc910dfb8937f72994234c95da0636ce9a82d48d28698bdf"}
Sep 16 12:13:24.129037 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:24.129015    2582 generic.go:356] "Generic (PLEG): container finished" podID="bc5cae68-af62-4c78-84e3-52d22ee21b53" containerID="860a51dc43aed15e0a71a1d87cc157cd0776816c812b076dcfa449f10a5b736a" exitCode=0
Sep 16 12:13:24.129125 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:24.129083    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9" event={"ID":"bc5cae68-af62-4c78-84e3-52d22ee21b53","Type":"ContainerDied","Data":"860a51dc43aed15e0a71a1d87cc157cd0776816c812b076dcfa449f10a5b736a"}
Sep 16 12:13:25.136997 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:25.136912    2582 generic.go:356] "Generic (PLEG): container finished" podID="88278d5b-8511-4dbc-a32d-292a27f9c1bf" containerID="0e17f9033801836e1371f3d69232b52a5182738d02ad0d30135ce4a2a6048f06" exitCode=0
Sep 16 12:13:25.136997 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:25.136985    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5" event={"ID":"88278d5b-8511-4dbc-a32d-292a27f9c1bf","Type":"ContainerDied","Data":"0e17f9033801836e1371f3d69232b52a5182738d02ad0d30135ce4a2a6048f06"}
Sep 16 12:13:25.139054 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:25.139027    2582 generic.go:356] "Generic (PLEG): container finished" podID="f3103550-7364-4ee8-85b8-628b1bda8767" containerID="e133b8c64cd3a9ba9277358ff8752fcdcb2807c55184d0a256456017fdc67bcf" exitCode=0
Sep 16 12:13:25.139189 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:25.139110    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c" event={"ID":"f3103550-7364-4ee8-85b8-628b1bda8767","Type":"ContainerDied","Data":"e133b8c64cd3a9ba9277358ff8752fcdcb2807c55184d0a256456017fdc67bcf"}
Sep 16 12:13:25.141326 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:25.141305    2582 generic.go:356] "Generic (PLEG): container finished" podID="bc5cae68-af62-4c78-84e3-52d22ee21b53" containerID="73401eb5931a97458007ccf98e63b92450d84e59f3aeba9819aa856e7245707a" exitCode=0
Sep 16 12:13:25.141448 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:25.141369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9" event={"ID":"bc5cae68-af62-4c78-84e3-52d22ee21b53","Type":"ContainerDied","Data":"73401eb5931a97458007ccf98e63b92450d84e59f3aeba9819aa856e7245707a"}
Sep 16 12:13:25.274723 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:25.274696    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj"
Sep 16 12:13:25.356765 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:25.356706    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b1122d0-9c42-4878-ba89-6fee21c3dac5-util\") pod \"1b1122d0-9c42-4878-ba89-6fee21c3dac5\" (UID: \"1b1122d0-9c42-4878-ba89-6fee21c3dac5\") "
Sep 16 12:13:25.356954 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:25.356801    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b1122d0-9c42-4878-ba89-6fee21c3dac5-bundle\") pod \"1b1122d0-9c42-4878-ba89-6fee21c3dac5\" (UID: \"1b1122d0-9c42-4878-ba89-6fee21c3dac5\") "
Sep 16 12:13:25.356954 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:25.356847    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2hg9\" (UniqueName: \"kubernetes.io/projected/1b1122d0-9c42-4878-ba89-6fee21c3dac5-kube-api-access-w2hg9\") pod \"1b1122d0-9c42-4878-ba89-6fee21c3dac5\" (UID: \"1b1122d0-9c42-4878-ba89-6fee21c3dac5\") "
Sep 16 12:13:25.357404 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:25.357371    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1122d0-9c42-4878-ba89-6fee21c3dac5-bundle" (OuterVolumeSpecName: "bundle") pod "1b1122d0-9c42-4878-ba89-6fee21c3dac5" (UID: "1b1122d0-9c42-4878-ba89-6fee21c3dac5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:13:25.359577 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:25.359550    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1122d0-9c42-4878-ba89-6fee21c3dac5-kube-api-access-w2hg9" (OuterVolumeSpecName: "kube-api-access-w2hg9") pod "1b1122d0-9c42-4878-ba89-6fee21c3dac5" (UID: "1b1122d0-9c42-4878-ba89-6fee21c3dac5"). InnerVolumeSpecName "kube-api-access-w2hg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:13:25.363432 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:25.363394    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1122d0-9c42-4878-ba89-6fee21c3dac5-util" (OuterVolumeSpecName: "util") pod "1b1122d0-9c42-4878-ba89-6fee21c3dac5" (UID: "1b1122d0-9c42-4878-ba89-6fee21c3dac5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:13:25.458011 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:25.457931    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b1122d0-9c42-4878-ba89-6fee21c3dac5-util\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:13:25.458011 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:25.457958    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b1122d0-9c42-4878-ba89-6fee21c3dac5-bundle\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:13:25.458011 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:25.457967    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2hg9\" (UniqueName: \"kubernetes.io/projected/1b1122d0-9c42-4878-ba89-6fee21c3dac5-kube-api-access-w2hg9\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:13:26.152607 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.152576    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj"
Sep 16 12:13:26.152607 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.152577    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxqhzj" event={"ID":"1b1122d0-9c42-4878-ba89-6fee21c3dac5","Type":"ContainerDied","Data":"f84929a9e76bae54c4327751320d21475dc61e2dd974e99d1c4482a637d24eea"}
Sep 16 12:13:26.153042 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.152633    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f84929a9e76bae54c4327751320d21475dc61e2dd974e99d1c4482a637d24eea"
Sep 16 12:13:26.304922 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.304889    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c"
Sep 16 12:13:26.353073 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.353049    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9"
Sep 16 12:13:26.356166 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.356146    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5"
Sep 16 12:13:26.465662 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.465557    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88278d5b-8511-4dbc-a32d-292a27f9c1bf-util\") pod \"88278d5b-8511-4dbc-a32d-292a27f9c1bf\" (UID: \"88278d5b-8511-4dbc-a32d-292a27f9c1bf\") "
Sep 16 12:13:26.465662 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.465627    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjz9q\" (UniqueName: \"kubernetes.io/projected/f3103550-7364-4ee8-85b8-628b1bda8767-kube-api-access-xjz9q\") pod \"f3103550-7364-4ee8-85b8-628b1bda8767\" (UID: \"f3103550-7364-4ee8-85b8-628b1bda8767\") "
Sep 16 12:13:26.465662 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.465663    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88278d5b-8511-4dbc-a32d-292a27f9c1bf-bundle\") pod \"88278d5b-8511-4dbc-a32d-292a27f9c1bf\" (UID: \"88278d5b-8511-4dbc-a32d-292a27f9c1bf\") "
Sep 16 12:13:26.465977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.465694    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s99pp\" (UniqueName: \"kubernetes.io/projected/bc5cae68-af62-4c78-84e3-52d22ee21b53-kube-api-access-s99pp\") pod \"bc5cae68-af62-4c78-84e3-52d22ee21b53\" (UID: \"bc5cae68-af62-4c78-84e3-52d22ee21b53\") "
Sep 16 12:13:26.465977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.465723    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3103550-7364-4ee8-85b8-628b1bda8767-bundle\") pod \"f3103550-7364-4ee8-85b8-628b1bda8767\" (UID: \"f3103550-7364-4ee8-85b8-628b1bda8767\") "
Sep 16 12:13:26.465977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.465809    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fd4ls\" (UniqueName: \"kubernetes.io/projected/88278d5b-8511-4dbc-a32d-292a27f9c1bf-kube-api-access-fd4ls\") pod \"88278d5b-8511-4dbc-a32d-292a27f9c1bf\" (UID: \"88278d5b-8511-4dbc-a32d-292a27f9c1bf\") "
Sep 16 12:13:26.465977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.465835    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc5cae68-af62-4c78-84e3-52d22ee21b53-util\") pod \"bc5cae68-af62-4c78-84e3-52d22ee21b53\" (UID: \"bc5cae68-af62-4c78-84e3-52d22ee21b53\") "
Sep 16 12:13:26.465977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.465856    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc5cae68-af62-4c78-84e3-52d22ee21b53-bundle\") pod \"bc5cae68-af62-4c78-84e3-52d22ee21b53\" (UID: \"bc5cae68-af62-4c78-84e3-52d22ee21b53\") "
Sep 16 12:13:26.465977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.465877    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3103550-7364-4ee8-85b8-628b1bda8767-util\") pod \"f3103550-7364-4ee8-85b8-628b1bda8767\" (UID: \"f3103550-7364-4ee8-85b8-628b1bda8767\") "
Sep 16 12:13:26.466589 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.466271    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88278d5b-8511-4dbc-a32d-292a27f9c1bf-bundle" (OuterVolumeSpecName: "bundle") pod "88278d5b-8511-4dbc-a32d-292a27f9c1bf" (UID: "88278d5b-8511-4dbc-a32d-292a27f9c1bf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:13:26.467179 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.467126    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5cae68-af62-4c78-84e3-52d22ee21b53-bundle" (OuterVolumeSpecName: "bundle") pod "bc5cae68-af62-4c78-84e3-52d22ee21b53" (UID: "bc5cae68-af62-4c78-84e3-52d22ee21b53"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:13:26.467370 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.467329    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3103550-7364-4ee8-85b8-628b1bda8767-bundle" (OuterVolumeSpecName: "bundle") pod "f3103550-7364-4ee8-85b8-628b1bda8767" (UID: "f3103550-7364-4ee8-85b8-628b1bda8767"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:13:26.468384 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.468364    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3103550-7364-4ee8-85b8-628b1bda8767-kube-api-access-xjz9q" (OuterVolumeSpecName: "kube-api-access-xjz9q") pod "f3103550-7364-4ee8-85b8-628b1bda8767" (UID: "f3103550-7364-4ee8-85b8-628b1bda8767"). InnerVolumeSpecName "kube-api-access-xjz9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:13:26.468579 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.468550    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88278d5b-8511-4dbc-a32d-292a27f9c1bf-kube-api-access-fd4ls" (OuterVolumeSpecName: "kube-api-access-fd4ls") pod "88278d5b-8511-4dbc-a32d-292a27f9c1bf" (UID: "88278d5b-8511-4dbc-a32d-292a27f9c1bf"). InnerVolumeSpecName "kube-api-access-fd4ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:13:26.468826 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.468799    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5cae68-af62-4c78-84e3-52d22ee21b53-kube-api-access-s99pp" (OuterVolumeSpecName: "kube-api-access-s99pp") pod "bc5cae68-af62-4c78-84e3-52d22ee21b53" (UID: "bc5cae68-af62-4c78-84e3-52d22ee21b53"). InnerVolumeSpecName "kube-api-access-s99pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:13:26.472680 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.472653    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88278d5b-8511-4dbc-a32d-292a27f9c1bf-util" (OuterVolumeSpecName: "util") pod "88278d5b-8511-4dbc-a32d-292a27f9c1bf" (UID: "88278d5b-8511-4dbc-a32d-292a27f9c1bf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:13:26.473120 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.473096    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5cae68-af62-4c78-84e3-52d22ee21b53-util" (OuterVolumeSpecName: "util") pod "bc5cae68-af62-4c78-84e3-52d22ee21b53" (UID: "bc5cae68-af62-4c78-84e3-52d22ee21b53"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:13:26.473268 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.473248    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3103550-7364-4ee8-85b8-628b1bda8767-util" (OuterVolumeSpecName: "util") pod "f3103550-7364-4ee8-85b8-628b1bda8767" (UID: "f3103550-7364-4ee8-85b8-628b1bda8767"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:13:26.566912 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.566876    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd4ls\" (UniqueName: \"kubernetes.io/projected/88278d5b-8511-4dbc-a32d-292a27f9c1bf-kube-api-access-fd4ls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:13:26.566912 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.566905    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc5cae68-af62-4c78-84e3-52d22ee21b53-util\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:13:26.566912 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.566917    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc5cae68-af62-4c78-84e3-52d22ee21b53-bundle\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:13:26.567137 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.566925    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3103550-7364-4ee8-85b8-628b1bda8767-util\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:13:26.567137 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.566932    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88278d5b-8511-4dbc-a32d-292a27f9c1bf-util\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:13:26.567137 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.566940    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjz9q\" (UniqueName: \"kubernetes.io/projected/f3103550-7364-4ee8-85b8-628b1bda8767-kube-api-access-xjz9q\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:13:26.567137 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.566949    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88278d5b-8511-4dbc-a32d-292a27f9c1bf-bundle\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:13:26.567137 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.566958    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s99pp\" (UniqueName: \"kubernetes.io/projected/bc5cae68-af62-4c78-84e3-52d22ee21b53-kube-api-access-s99pp\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:13:26.567137 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:26.566965    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3103550-7364-4ee8-85b8-628b1bda8767-bundle\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:13:27.159790 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:27.159713    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9"
Sep 16 12:13:27.159790 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:27.159768    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2wsz9" event={"ID":"bc5cae68-af62-4c78-84e3-52d22ee21b53","Type":"ContainerDied","Data":"7f2944f2353346e89e2da8f78fd2057b1205e082921657113da05d85a3f55aa3"}
Sep 16 12:13:27.160293 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:27.159863    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f2944f2353346e89e2da8f78fd2057b1205e082921657113da05d85a3f55aa3"
Sep 16 12:13:27.161875 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:27.161846    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5" event={"ID":"88278d5b-8511-4dbc-a32d-292a27f9c1bf","Type":"ContainerDied","Data":"af582349dafe3ea4ba308e8fcd98e8233ad50884ceaa37bf81d7ec2d89ff3673"}
Sep 16 12:13:27.162004 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:27.161883    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c622bg5"
Sep 16 12:13:27.162004 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:27.161901    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af582349dafe3ea4ba308e8fcd98e8233ad50884ceaa37bf81d7ec2d89ff3673"
Sep 16 12:13:27.163750 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:27.163711    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c" event={"ID":"f3103550-7364-4ee8-85b8-628b1bda8767","Type":"ContainerDied","Data":"7da06f8b2684dc016661941ecbbd7f91e9bc57752db04cfecdd39ed0a9e2310d"}
Sep 16 12:13:27.163864 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:27.163763    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7da06f8b2684dc016661941ecbbd7f91e9bc57752db04cfecdd39ed0a9e2310d"
Sep 16 12:13:27.163864 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:27.163721    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l9w4c"
Sep 16 12:13:47.520604 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.520567    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb"]
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.520892    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3103550-7364-4ee8-85b8-628b1bda8767" containerName="pull"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.520903    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3103550-7364-4ee8-85b8-628b1bda8767" containerName="pull"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.520918    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc5cae68-af62-4c78-84e3-52d22ee21b53" containerName="util"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.520924    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc5cae68-af62-4c78-84e3-52d22ee21b53" containerName="util"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.520930    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88278d5b-8511-4dbc-a32d-292a27f9c1bf" containerName="extract"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.520935    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="88278d5b-8511-4dbc-a32d-292a27f9c1bf" containerName="extract"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.520943    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b1122d0-9c42-4878-ba89-6fee21c3dac5" containerName="util"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.520948    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b1122d0-9c42-4878-ba89-6fee21c3dac5" containerName="util"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.520954    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b1122d0-9c42-4878-ba89-6fee21c3dac5" containerName="extract"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.520959    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b1122d0-9c42-4878-ba89-6fee21c3dac5" containerName="extract"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.520965    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88278d5b-8511-4dbc-a32d-292a27f9c1bf" containerName="pull"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.520969    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="88278d5b-8511-4dbc-a32d-292a27f9c1bf" containerName="pull"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.520977    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc5cae68-af62-4c78-84e3-52d22ee21b53" containerName="pull"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.520981    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc5cae68-af62-4c78-84e3-52d22ee21b53" containerName="pull"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.520990    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3103550-7364-4ee8-85b8-628b1bda8767" containerName="util"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.520994    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3103550-7364-4ee8-85b8-628b1bda8767" containerName="util"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.521003    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88278d5b-8511-4dbc-a32d-292a27f9c1bf" containerName="util"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.521008    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="88278d5b-8511-4dbc-a32d-292a27f9c1bf" containerName="util"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.521013    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc5cae68-af62-4c78-84e3-52d22ee21b53" containerName="extract"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.521018    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc5cae68-af62-4c78-84e3-52d22ee21b53" containerName="extract"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.521026    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b1122d0-9c42-4878-ba89-6fee21c3dac5" containerName="pull"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.521031    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b1122d0-9c42-4878-ba89-6fee21c3dac5" containerName="pull"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.521035    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3103550-7364-4ee8-85b8-628b1bda8767" containerName="extract"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.521041    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3103550-7364-4ee8-85b8-628b1bda8767" containerName="extract"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.521090    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b1122d0-9c42-4878-ba89-6fee21c3dac5" containerName="extract"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.521097    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc5cae68-af62-4c78-84e3-52d22ee21b53" containerName="extract"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.521103    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3103550-7364-4ee8-85b8-628b1bda8767" containerName="extract"
Sep 16 12:13:47.521131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.521110    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="88278d5b-8511-4dbc-a32d-292a27f9c1bf" containerName="extract"
Sep 16 12:13:47.524650 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.524630    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb"
Sep 16 12:13:47.528261 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.528235    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 12:13:47.528424 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.528292    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 12:13:47.528424 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.528320    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 12:13:47.529327 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.529306    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 12:13:47.529438 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.529340    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bm6lh\""
Sep 16 12:13:47.534505 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.534486    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb"]
Sep 16 12:13:47.545800 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.545771    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v6rf\" (UniqueName: \"kubernetes.io/projected/d39938d5-6891-4ee8-b7e4-85452ea78911-kube-api-access-6v6rf\") pod \"kuadrant-console-plugin-fbdd4f445-x6llb\" (UID: \"d39938d5-6891-4ee8-b7e4-85452ea78911\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb"
Sep 16 12:13:47.545931 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.545887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d39938d5-6891-4ee8-b7e4-85452ea78911-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-x6llb\" (UID: \"d39938d5-6891-4ee8-b7e4-85452ea78911\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb"
Sep 16 12:13:47.546002 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.545931    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d39938d5-6891-4ee8-b7e4-85452ea78911-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-x6llb\" (UID: \"d39938d5-6891-4ee8-b7e4-85452ea78911\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb"
Sep 16 12:13:47.646886 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.646851    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d39938d5-6891-4ee8-b7e4-85452ea78911-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-x6llb\" (UID: \"d39938d5-6891-4ee8-b7e4-85452ea78911\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb"
Sep 16 12:13:47.647053 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.646895    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d39938d5-6891-4ee8-b7e4-85452ea78911-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-x6llb\" (UID: \"d39938d5-6891-4ee8-b7e4-85452ea78911\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb"
Sep 16 12:13:47.647053 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.646921    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v6rf\" (UniqueName: \"kubernetes.io/projected/d39938d5-6891-4ee8-b7e4-85452ea78911-kube-api-access-6v6rf\") pod \"kuadrant-console-plugin-fbdd4f445-x6llb\" (UID: \"d39938d5-6891-4ee8-b7e4-85452ea78911\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb"
Sep 16 12:13:47.647053 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:13:47.647005    2582 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 16 12:13:47.647165 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:13:47.647080    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d39938d5-6891-4ee8-b7e4-85452ea78911-plugin-serving-cert podName:d39938d5-6891-4ee8-b7e4-85452ea78911 nodeName:}" failed. No retries permitted until 2026-09-16 12:13:48.147062854 +0000 UTC m=+787.397600383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/d39938d5-6891-4ee8-b7e4-85452ea78911-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-x6llb" (UID: "d39938d5-6891-4ee8-b7e4-85452ea78911") : secret "plugin-serving-cert" not found
Sep 16 12:13:47.647520 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.647501    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d39938d5-6891-4ee8-b7e4-85452ea78911-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-x6llb\" (UID: \"d39938d5-6891-4ee8-b7e4-85452ea78911\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb"
Sep 16 12:13:47.655714 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:47.655688    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v6rf\" (UniqueName: \"kubernetes.io/projected/d39938d5-6891-4ee8-b7e4-85452ea78911-kube-api-access-6v6rf\") pod \"kuadrant-console-plugin-fbdd4f445-x6llb\" (UID: \"d39938d5-6891-4ee8-b7e4-85452ea78911\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb"
Sep 16 12:13:48.151192 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:48.151147    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d39938d5-6891-4ee8-b7e4-85452ea78911-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-x6llb\" (UID: \"d39938d5-6891-4ee8-b7e4-85452ea78911\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb"
Sep 16 12:13:48.151395 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:13:48.151268    2582 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 16 12:13:48.151395 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:13:48.151335    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d39938d5-6891-4ee8-b7e4-85452ea78911-plugin-serving-cert podName:d39938d5-6891-4ee8-b7e4-85452ea78911 nodeName:}" failed. No retries permitted until 2026-09-16 12:13:49.151318027 +0000 UTC m=+788.401855559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/d39938d5-6891-4ee8-b7e4-85452ea78911-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-x6llb" (UID: "d39938d5-6891-4ee8-b7e4-85452ea78911") : secret "plugin-serving-cert" not found
Sep 16 12:13:49.158883 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:49.158848    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d39938d5-6891-4ee8-b7e4-85452ea78911-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-x6llb\" (UID: \"d39938d5-6891-4ee8-b7e4-85452ea78911\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb"
Sep 16 12:13:49.163241 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:49.163207    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d39938d5-6891-4ee8-b7e4-85452ea78911-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-x6llb\" (UID: \"d39938d5-6891-4ee8-b7e4-85452ea78911\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb"
Sep 16 12:13:49.334012 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:49.333978    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb"
Sep 16 12:13:49.605136 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:49.605107    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb"]
Sep 16 12:13:49.606663 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:13:49.606631    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd39938d5_6891_4ee8_b7e4_85452ea78911.slice/crio-3822d78dd90ad10f002419242c01c73f074fbb120ca85ce75dd5e1666c09049b WatchSource:0}: Error finding container 3822d78dd90ad10f002419242c01c73f074fbb120ca85ce75dd5e1666c09049b: Status 404 returned error can't find the container with id 3822d78dd90ad10f002419242c01c73f074fbb120ca85ce75dd5e1666c09049b
Sep 16 12:13:50.277797 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:50.277750    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb" event={"ID":"d39938d5-6891-4ee8-b7e4-85452ea78911","Type":"ContainerStarted","Data":"3822d78dd90ad10f002419242c01c73f074fbb120ca85ce75dd5e1666c09049b"}
Sep 16 12:13:55.306574 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:55.306535    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb" event={"ID":"d39938d5-6891-4ee8-b7e4-85452ea78911","Type":"ContainerStarted","Data":"c774a2d145f79c16d9f51029caf35d758de389f5466aeed61ac849b4a4a2000d"}
Sep 16 12:13:55.323017 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:13:55.322966    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x6llb" podStartSLOduration=3.473407454 podStartE2EDuration="8.322950398s" totalImagesPullingTime="4.849542944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:13:55.322042719 +0000 UTC m=+794.572580268" watchObservedRunningTime="2026-09-16 12:13:55.322950398 +0000 UTC m=+794.573487947"
Sep 16 12:14:40.390178 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:40.390146    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rw7kk"]
Sep 16 12:14:40.393681 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:40.393656    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-rw7kk"
Sep 16 12:14:40.396089 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:40.396065    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4xxz4\""
Sep 16 12:14:40.401278 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:40.401249    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rw7kk"]
Sep 16 12:14:40.406543 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:40.406517    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cwcs\" (UniqueName: \"kubernetes.io/projected/01609880-33db-43b5-93eb-c1b2edc93bc1-kube-api-access-5cwcs\") pod \"authorino-7bcfd8f9cd-rw7kk\" (UID: \"01609880-33db-43b5-93eb-c1b2edc93bc1\") " pod="kuadrant-system/authorino-7bcfd8f9cd-rw7kk"
Sep 16 12:14:40.507679 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:40.507638    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cwcs\" (UniqueName: \"kubernetes.io/projected/01609880-33db-43b5-93eb-c1b2edc93bc1-kube-api-access-5cwcs\") pod \"authorino-7bcfd8f9cd-rw7kk\" (UID: \"01609880-33db-43b5-93eb-c1b2edc93bc1\") " pod="kuadrant-system/authorino-7bcfd8f9cd-rw7kk"
Sep 16 12:14:40.516289 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:40.516256    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cwcs\" (UniqueName: \"kubernetes.io/projected/01609880-33db-43b5-93eb-c1b2edc93bc1-kube-api-access-5cwcs\") pod \"authorino-7bcfd8f9cd-rw7kk\" (UID: \"01609880-33db-43b5-93eb-c1b2edc93bc1\") " pod="kuadrant-system/authorino-7bcfd8f9cd-rw7kk"
Sep 16 12:14:40.704273 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:40.704182    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-rw7kk"
Sep 16 12:14:40.974035 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:40.974011    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rw7kk"]
Sep 16 12:14:40.975720 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:14:40.975693    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01609880_33db_43b5_93eb_c1b2edc93bc1.slice/crio-ec299a5465b6ecda2dd900ff46f6d593d03c938e040383cdee2b1901e1bf748c WatchSource:0}: Error finding container ec299a5465b6ecda2dd900ff46f6d593d03c938e040383cdee2b1901e1bf748c: Status 404 returned error can't find the container with id ec299a5465b6ecda2dd900ff46f6d593d03c938e040383cdee2b1901e1bf748c
Sep 16 12:14:41.549464 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:41.549430    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-rw7kk" event={"ID":"01609880-33db-43b5-93eb-c1b2edc93bc1","Type":"ContainerStarted","Data":"ec299a5465b6ecda2dd900ff46f6d593d03c938e040383cdee2b1901e1bf748c"}
Sep 16 12:14:43.167517 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:43.167482    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rw7kk"]
Sep 16 12:14:43.561840 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:43.561801    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-rw7kk" event={"ID":"01609880-33db-43b5-93eb-c1b2edc93bc1","Type":"ContainerStarted","Data":"2822bb15a8a08cde9c098a15042ce1cd8f0c602467b0782601be22eeffc54825"}
Sep 16 12:14:43.561840 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:43.561804    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-rw7kk" podUID="01609880-33db-43b5-93eb-c1b2edc93bc1" containerName="authorino" containerID="cri-o://2822bb15a8a08cde9c098a15042ce1cd8f0c602467b0782601be22eeffc54825" gracePeriod=30
Sep 16 12:14:43.580481 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:43.580439    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-rw7kk" podStartSLOduration=1.082879524 podStartE2EDuration="3.580421187s" totalImagesPullingTime="2.497541663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:43.579689885 +0000 UTC m=+842.830227435" watchObservedRunningTime="2026-09-16 12:14:43.580421187 +0000 UTC m=+842.830958790"
Sep 16 12:14:43.805074 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:43.805039    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-rw7kk"
Sep 16 12:14:43.836713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:43.836686    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cwcs\" (UniqueName: \"kubernetes.io/projected/01609880-33db-43b5-93eb-c1b2edc93bc1-kube-api-access-5cwcs\") pod \"01609880-33db-43b5-93eb-c1b2edc93bc1\" (UID: \"01609880-33db-43b5-93eb-c1b2edc93bc1\") "
Sep 16 12:14:43.839048 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:43.839014    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01609880-33db-43b5-93eb-c1b2edc93bc1-kube-api-access-5cwcs" (OuterVolumeSpecName: "kube-api-access-5cwcs") pod "01609880-33db-43b5-93eb-c1b2edc93bc1" (UID: "01609880-33db-43b5-93eb-c1b2edc93bc1"). InnerVolumeSpecName "kube-api-access-5cwcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:14:43.937886 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:43.937848    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cwcs\" (UniqueName: \"kubernetes.io/projected/01609880-33db-43b5-93eb-c1b2edc93bc1-kube-api-access-5cwcs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:14:44.567833 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:44.567800    2582 generic.go:356] "Generic (PLEG): container finished" podID="01609880-33db-43b5-93eb-c1b2edc93bc1" containerID="2822bb15a8a08cde9c098a15042ce1cd8f0c602467b0782601be22eeffc54825" exitCode=2
Sep 16 12:14:44.568329 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:44.567858    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-rw7kk"
Sep 16 12:14:44.568329 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:44.567870    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-rw7kk" event={"ID":"01609880-33db-43b5-93eb-c1b2edc93bc1","Type":"ContainerDied","Data":"2822bb15a8a08cde9c098a15042ce1cd8f0c602467b0782601be22eeffc54825"}
Sep 16 12:14:44.568329 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:44.567901    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-rw7kk" event={"ID":"01609880-33db-43b5-93eb-c1b2edc93bc1","Type":"ContainerDied","Data":"ec299a5465b6ecda2dd900ff46f6d593d03c938e040383cdee2b1901e1bf748c"}
Sep 16 12:14:44.568329 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:44.567917    2582 scope.go:117] "RemoveContainer" containerID="2822bb15a8a08cde9c098a15042ce1cd8f0c602467b0782601be22eeffc54825"
Sep 16 12:14:44.577116 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:44.577092    2582 scope.go:117] "RemoveContainer" containerID="2822bb15a8a08cde9c098a15042ce1cd8f0c602467b0782601be22eeffc54825"
Sep 16 12:14:44.577424 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:14:44.577402    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2822bb15a8a08cde9c098a15042ce1cd8f0c602467b0782601be22eeffc54825\": container with ID starting with 2822bb15a8a08cde9c098a15042ce1cd8f0c602467b0782601be22eeffc54825 not found: ID does not exist" containerID="2822bb15a8a08cde9c098a15042ce1cd8f0c602467b0782601be22eeffc54825"
Sep 16 12:14:44.577489 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:44.577437    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2822bb15a8a08cde9c098a15042ce1cd8f0c602467b0782601be22eeffc54825"} err="failed to get container status \"2822bb15a8a08cde9c098a15042ce1cd8f0c602467b0782601be22eeffc54825\": rpc error: code = NotFound desc = could not find container \"2822bb15a8a08cde9c098a15042ce1cd8f0c602467b0782601be22eeffc54825\": container with ID starting with 2822bb15a8a08cde9c098a15042ce1cd8f0c602467b0782601be22eeffc54825 not found: ID does not exist"
Sep 16 12:14:44.588688 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:44.588662    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rw7kk"]
Sep 16 12:14:44.592115 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:44.592091    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rw7kk"]
Sep 16 12:14:45.212326 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:14:45.212292    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01609880-33db-43b5-93eb-c1b2edc93bc1" path="/var/lib/kubelet/pods/01609880-33db-43b5-93eb-c1b2edc93bc1/volumes"
Sep 16 12:15:08.844558 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:08.844522    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-b9wt9"]
Sep 16 12:15:08.845040 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:08.844867    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01609880-33db-43b5-93eb-c1b2edc93bc1" containerName="authorino"
Sep 16 12:15:08.845040 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:08.844881    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="01609880-33db-43b5-93eb-c1b2edc93bc1" containerName="authorino"
Sep 16 12:15:08.845040 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:08.844950    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="01609880-33db-43b5-93eb-c1b2edc93bc1" containerName="authorino"
Sep 16 12:15:08.847905 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:08.847887    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-b9wt9"
Sep 16 12:15:08.850540 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:08.850518    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4xxz4\""
Sep 16 12:15:08.855101 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:08.855075    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-b9wt9"]
Sep 16 12:15:08.897340 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:08.897297    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-b9wt9"]
Sep 16 12:15:08.897607 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:15:08.897537    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-brmfv], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-b9wt9" podUID="5bd810a2-42fa-4537-8463-492be6937934"
Sep 16 12:15:08.932922 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:08.932885    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9nltj"]
Sep 16 12:15:08.936247 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:08.936231    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9nltj"
Sep 16 12:15:08.938965 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:08.938944    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 12:15:08.946580 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:08.946555    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9nltj"]
Sep 16 12:15:08.959422 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:08.959399    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brmfv\" (UniqueName: \"kubernetes.io/projected/5bd810a2-42fa-4537-8463-492be6937934-kube-api-access-brmfv\") pod \"authorino-557f876f69-b9wt9\" (UID: \"5bd810a2-42fa-4537-8463-492be6937934\") " pod="kuadrant-system/authorino-557f876f69-b9wt9"
Sep 16 12:15:09.060776 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.060716    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0823d33-7c13-419c-bbbd-2fedb4fb6606-tls-cert\") pod \"authorino-6d4dfc649f-9nltj\" (UID: \"f0823d33-7c13-419c-bbbd-2fedb4fb6606\") " pod="kuadrant-system/authorino-6d4dfc649f-9nltj"
Sep 16 12:15:09.060947 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.060855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwzvc\" (UniqueName: \"kubernetes.io/projected/f0823d33-7c13-419c-bbbd-2fedb4fb6606-kube-api-access-hwzvc\") pod \"authorino-6d4dfc649f-9nltj\" (UID: \"f0823d33-7c13-419c-bbbd-2fedb4fb6606\") " pod="kuadrant-system/authorino-6d4dfc649f-9nltj"
Sep 16 12:15:09.060947 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.060908    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brmfv\" (UniqueName: \"kubernetes.io/projected/5bd810a2-42fa-4537-8463-492be6937934-kube-api-access-brmfv\") pod \"authorino-557f876f69-b9wt9\" (UID: \"5bd810a2-42fa-4537-8463-492be6937934\") " pod="kuadrant-system/authorino-557f876f69-b9wt9"
Sep 16 12:15:09.069631 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.069607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brmfv\" (UniqueName: \"kubernetes.io/projected/5bd810a2-42fa-4537-8463-492be6937934-kube-api-access-brmfv\") pod \"authorino-557f876f69-b9wt9\" (UID: \"5bd810a2-42fa-4537-8463-492be6937934\") " pod="kuadrant-system/authorino-557f876f69-b9wt9"
Sep 16 12:15:09.095464 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.095386    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9nltj"]
Sep 16 12:15:09.095656 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:15:09.095636    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hwzvc tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-9nltj" podUID="f0823d33-7c13-419c-bbbd-2fedb4fb6606"
Sep 16 12:15:09.130494 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.130461    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79f8797474-zndjv"]
Sep 16 12:15:09.141703 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.133912    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79f8797474-zndjv"
Sep 16 12:15:09.144919 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.144893    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79f8797474-zndjv"]
Sep 16 12:15:09.162250 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.162213    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0823d33-7c13-419c-bbbd-2fedb4fb6606-tls-cert\") pod \"authorino-6d4dfc649f-9nltj\" (UID: \"f0823d33-7c13-419c-bbbd-2fedb4fb6606\") " pod="kuadrant-system/authorino-6d4dfc649f-9nltj"
Sep 16 12:15:09.162418 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.162293    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwzvc\" (UniqueName: \"kubernetes.io/projected/f0823d33-7c13-419c-bbbd-2fedb4fb6606-kube-api-access-hwzvc\") pod \"authorino-6d4dfc649f-9nltj\" (UID: \"f0823d33-7c13-419c-bbbd-2fedb4fb6606\") " pod="kuadrant-system/authorino-6d4dfc649f-9nltj"
Sep 16 12:15:09.164819 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.164798    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0823d33-7c13-419c-bbbd-2fedb4fb6606-tls-cert\") pod \"authorino-6d4dfc649f-9nltj\" (UID: \"f0823d33-7c13-419c-bbbd-2fedb4fb6606\") " pod="kuadrant-system/authorino-6d4dfc649f-9nltj"
Sep 16 12:15:09.170563 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.170544    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwzvc\" (UniqueName: \"kubernetes.io/projected/f0823d33-7c13-419c-bbbd-2fedb4fb6606-kube-api-access-hwzvc\") pod \"authorino-6d4dfc649f-9nltj\" (UID: \"f0823d33-7c13-419c-bbbd-2fedb4fb6606\") " pod="kuadrant-system/authorino-6d4dfc649f-9nltj"
Sep 16 12:15:09.263482 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.263446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml7lq\" (UniqueName: \"kubernetes.io/projected/964f5922-be1c-4f28-b064-976884195de2-kube-api-access-ml7lq\") pod \"authorino-79f8797474-zndjv\" (UID: \"964f5922-be1c-4f28-b064-976884195de2\") " pod="kuadrant-system/authorino-79f8797474-zndjv"
Sep 16 12:15:09.263482 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.263488    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/964f5922-be1c-4f28-b064-976884195de2-tls-cert\") pod \"authorino-79f8797474-zndjv\" (UID: \"964f5922-be1c-4f28-b064-976884195de2\") " pod="kuadrant-system/authorino-79f8797474-zndjv"
Sep 16 12:15:09.364111 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.364007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml7lq\" (UniqueName: \"kubernetes.io/projected/964f5922-be1c-4f28-b064-976884195de2-kube-api-access-ml7lq\") pod \"authorino-79f8797474-zndjv\" (UID: \"964f5922-be1c-4f28-b064-976884195de2\") " pod="kuadrant-system/authorino-79f8797474-zndjv"
Sep 16 12:15:09.364111 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.364061    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/964f5922-be1c-4f28-b064-976884195de2-tls-cert\") pod \"authorino-79f8797474-zndjv\" (UID: \"964f5922-be1c-4f28-b064-976884195de2\") " pod="kuadrant-system/authorino-79f8797474-zndjv"
Sep 16 12:15:09.366600 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.366577    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/964f5922-be1c-4f28-b064-976884195de2-tls-cert\") pod \"authorino-79f8797474-zndjv\" (UID: \"964f5922-be1c-4f28-b064-976884195de2\") " pod="kuadrant-system/authorino-79f8797474-zndjv"
Sep 16 12:15:09.372196 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.372170    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml7lq\" (UniqueName: \"kubernetes.io/projected/964f5922-be1c-4f28-b064-976884195de2-kube-api-access-ml7lq\") pod \"authorino-79f8797474-zndjv\" (UID: \"964f5922-be1c-4f28-b064-976884195de2\") " pod="kuadrant-system/authorino-79f8797474-zndjv"
Sep 16 12:15:09.452167 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.452130    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79f8797474-zndjv"
Sep 16 12:15:09.692963 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.692877    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9nltj"
Sep 16 12:15:09.692963 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.692892    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-b9wt9"
Sep 16 12:15:09.700858 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.700831    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9nltj"
Sep 16 12:15:09.703364 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.703349    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-b9wt9"
Sep 16 12:15:09.724677 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.724646    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79f8797474-zndjv"]
Sep 16 12:15:09.727116 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:15:09.727090    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod964f5922_be1c_4f28_b064_976884195de2.slice/crio-f6aa74ce40c4eac849b27ed9965605c0b946a57be65a22531e41608150e1832c WatchSource:0}: Error finding container f6aa74ce40c4eac849b27ed9965605c0b946a57be65a22531e41608150e1832c: Status 404 returned error can't find the container with id f6aa74ce40c4eac849b27ed9965605c0b946a57be65a22531e41608150e1832c
Sep 16 12:15:09.868651 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.868618    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwzvc\" (UniqueName: \"kubernetes.io/projected/f0823d33-7c13-419c-bbbd-2fedb4fb6606-kube-api-access-hwzvc\") pod \"f0823d33-7c13-419c-bbbd-2fedb4fb6606\" (UID: \"f0823d33-7c13-419c-bbbd-2fedb4fb6606\") "
Sep 16 12:15:09.869148 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.868682    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0823d33-7c13-419c-bbbd-2fedb4fb6606-tls-cert\") pod \"f0823d33-7c13-419c-bbbd-2fedb4fb6606\" (UID: \"f0823d33-7c13-419c-bbbd-2fedb4fb6606\") "
Sep 16 12:15:09.869148 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.868710    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brmfv\" (UniqueName: \"kubernetes.io/projected/5bd810a2-42fa-4537-8463-492be6937934-kube-api-access-brmfv\") pod \"5bd810a2-42fa-4537-8463-492be6937934\" (UID: \"5bd810a2-42fa-4537-8463-492be6937934\") "
Sep 16 12:15:09.871084 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.871054    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd810a2-42fa-4537-8463-492be6937934-kube-api-access-brmfv" (OuterVolumeSpecName: "kube-api-access-brmfv") pod "5bd810a2-42fa-4537-8463-492be6937934" (UID: "5bd810a2-42fa-4537-8463-492be6937934"). InnerVolumeSpecName "kube-api-access-brmfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:15:09.871193 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.871057    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0823d33-7c13-419c-bbbd-2fedb4fb6606-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f0823d33-7c13-419c-bbbd-2fedb4fb6606" (UID: "f0823d33-7c13-419c-bbbd-2fedb4fb6606"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:15:09.871193 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.871113    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0823d33-7c13-419c-bbbd-2fedb4fb6606-kube-api-access-hwzvc" (OuterVolumeSpecName: "kube-api-access-hwzvc") pod "f0823d33-7c13-419c-bbbd-2fedb4fb6606" (UID: "f0823d33-7c13-419c-bbbd-2fedb4fb6606"). InnerVolumeSpecName "kube-api-access-hwzvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:15:09.969409 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.969324    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brmfv\" (UniqueName: \"kubernetes.io/projected/5bd810a2-42fa-4537-8463-492be6937934-kube-api-access-brmfv\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:15:09.969409 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.969352    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwzvc\" (UniqueName: \"kubernetes.io/projected/f0823d33-7c13-419c-bbbd-2fedb4fb6606-kube-api-access-hwzvc\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:15:09.969409 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:09.969363    2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0823d33-7c13-419c-bbbd-2fedb4fb6606-tls-cert\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:15:10.701134 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:10.700904    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79f8797474-zndjv" event={"ID":"964f5922-be1c-4f28-b064-976884195de2","Type":"ContainerStarted","Data":"0b4198b88db99b6a3df7ded5e6bca0ffec670e7e4a8828ce9664aff86f43f64a"}
Sep 16 12:15:10.701134 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:10.700953    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79f8797474-zndjv" event={"ID":"964f5922-be1c-4f28-b064-976884195de2","Type":"ContainerStarted","Data":"f6aa74ce40c4eac849b27ed9965605c0b946a57be65a22531e41608150e1832c"}
Sep 16 12:15:10.701134 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:10.701001    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-b9wt9"
Sep 16 12:15:10.701134 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:10.701099    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9nltj"
Sep 16 12:15:10.720400 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:10.720346    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-79f8797474-zndjv" podStartSLOduration=0.836597792 podStartE2EDuration="1.720328786s" totalImagesPullingTime="883.730994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:10.718774298 +0000 UTC m=+869.969311845" watchObservedRunningTime="2026-09-16 12:15:10.720328786 +0000 UTC m=+869.970866334"
Sep 16 12:15:10.746670 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:10.746632    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-b9wt9"]
Sep 16 12:15:10.758309 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:10.758273    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-b9wt9"]
Sep 16 12:15:10.793873 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:10.793835    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9nltj"]
Sep 16 12:15:10.798091 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:10.798062    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9nltj"]
Sep 16 12:15:11.214659 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:11.214626    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bd810a2-42fa-4537-8463-492be6937934" path="/var/lib/kubelet/pods/5bd810a2-42fa-4537-8463-492be6937934/volumes"
Sep 16 12:15:11.215190 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:11.215021    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0823d33-7c13-419c-bbbd-2fedb4fb6606" path="/var/lib/kubelet/pods/f0823d33-7c13-419c-bbbd-2fedb4fb6606/volumes"
Sep 16 12:15:52.533496 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.533458    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"]
Sep 16 12:15:52.535610 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.535593    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:15:52.539519 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.539489    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:15:52.539656 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.539490    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:15:52.539656 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.539497    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 12:15:52.539656 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.539498    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:15:52.539656 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.539502    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wwdn8\""
Sep 16 12:15:52.546238 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.546208    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"]
Sep 16 12:15:52.622022 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.621977    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3499149c-92a9-4514-8dfb-f57bcd1b66b0-metrics-certs\") pod \"maas-api-74767d9c48-vdvqx\" (UID: \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\") " pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:15:52.622022 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.622030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3499149c-92a9-4514-8dfb-f57bcd1b66b0-maas-api-tls\") pod \"maas-api-74767d9c48-vdvqx\" (UID: \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\") " pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:15:52.622242 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.622122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml8gx\" (UniqueName: \"kubernetes.io/projected/3499149c-92a9-4514-8dfb-f57bcd1b66b0-kube-api-access-ml8gx\") pod \"maas-api-74767d9c48-vdvqx\" (UID: \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\") " pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:15:52.622242 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.622172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3499149c-92a9-4514-8dfb-f57bcd1b66b0-openshift-service-ca\") pod \"maas-api-74767d9c48-vdvqx\" (UID: \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\") " pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:15:52.723486 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.723444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3499149c-92a9-4514-8dfb-f57bcd1b66b0-metrics-certs\") pod \"maas-api-74767d9c48-vdvqx\" (UID: \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\") " pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:15:52.723697 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.723496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3499149c-92a9-4514-8dfb-f57bcd1b66b0-maas-api-tls\") pod \"maas-api-74767d9c48-vdvqx\" (UID: \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\") " pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:15:52.723697 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.723556    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml8gx\" (UniqueName: \"kubernetes.io/projected/3499149c-92a9-4514-8dfb-f57bcd1b66b0-kube-api-access-ml8gx\") pod \"maas-api-74767d9c48-vdvqx\" (UID: \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\") " pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:15:52.723697 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.723605    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3499149c-92a9-4514-8dfb-f57bcd1b66b0-openshift-service-ca\") pod \"maas-api-74767d9c48-vdvqx\" (UID: \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\") " pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:15:52.723697 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:15:52.723613    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 16 12:15:52.723697 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:15:52.723692    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3499149c-92a9-4514-8dfb-f57bcd1b66b0-metrics-certs podName:3499149c-92a9-4514-8dfb-f57bcd1b66b0 nodeName:}" failed. No retries permitted until 2026-09-16 12:15:53.223673213 +0000 UTC m=+912.474210740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3499149c-92a9-4514-8dfb-f57bcd1b66b0-metrics-certs") pod "maas-api-74767d9c48-vdvqx" (UID: "3499149c-92a9-4514-8dfb-f57bcd1b66b0") : secret "maas-api-metrics-tls" not found
Sep 16 12:15:52.724361 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.724338    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3499149c-92a9-4514-8dfb-f57bcd1b66b0-openshift-service-ca\") pod \"maas-api-74767d9c48-vdvqx\" (UID: \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\") " pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:15:52.726129 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.726109    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3499149c-92a9-4514-8dfb-f57bcd1b66b0-maas-api-tls\") pod \"maas-api-74767d9c48-vdvqx\" (UID: \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\") " pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:15:52.740446 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:52.740421    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml8gx\" (UniqueName: \"kubernetes.io/projected/3499149c-92a9-4514-8dfb-f57bcd1b66b0-kube-api-access-ml8gx\") pod \"maas-api-74767d9c48-vdvqx\" (UID: \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\") " pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:15:53.228125 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:53.228077    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3499149c-92a9-4514-8dfb-f57bcd1b66b0-metrics-certs\") pod \"maas-api-74767d9c48-vdvqx\" (UID: \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\") " pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:15:53.230716 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:53.230681    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3499149c-92a9-4514-8dfb-f57bcd1b66b0-metrics-certs\") pod \"maas-api-74767d9c48-vdvqx\" (UID: \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\") " pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:15:53.446337 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:53.446296    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:15:53.718566 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:53.718539    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"]
Sep 16 12:15:53.720455 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:15:53.720426    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3499149c_92a9_4514_8dfb_f57bcd1b66b0.slice/crio-28657cdfe17de5858127d3bfc1429748f7e8a7400646f1251e254cda9b46440c WatchSource:0}: Error finding container 28657cdfe17de5858127d3bfc1429748f7e8a7400646f1251e254cda9b46440c: Status 404 returned error can't find the container with id 28657cdfe17de5858127d3bfc1429748f7e8a7400646f1251e254cda9b46440c
Sep 16 12:15:53.918866 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:53.918823    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx" event={"ID":"3499149c-92a9-4514-8dfb-f57bcd1b66b0","Type":"ContainerStarted","Data":"28657cdfe17de5858127d3bfc1429748f7e8a7400646f1251e254cda9b46440c"}
Sep 16 12:15:56.939837 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:56.939797    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx" event={"ID":"3499149c-92a9-4514-8dfb-f57bcd1b66b0","Type":"ContainerStarted","Data":"d79da1d96e8de25317fdec5b086fbbf15b9e9f1d276934892a4208731502bc45"}
Sep 16 12:15:56.940370 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:56.939898    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:15:56.957948 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:15:56.957906    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx" podStartSLOduration=2.496329963 podStartE2EDuration="4.957891502s" totalImagesPullingTime="2.461561539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:56.955820137 +0000 UTC m=+916.206357678" watchObservedRunningTime="2026-09-16 12:15:56.957891502 +0000 UTC m=+916.208429048"
Sep 16 12:16:02.951147 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:16:02.951113    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:17:11.007677 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.007587    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"]
Sep 16 12:17:11.010886 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.010867    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.014932 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.014890    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 12:17:11.015076 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.014955    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jfdlt\""
Sep 16 12:17:11.015149 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.015136    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 12:17:11.015304 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.015287    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 16 12:17:11.020922 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.020900    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"]
Sep 16 12:17:11.110985 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.110951    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnzbt\" (UniqueName: \"kubernetes.io/projected/01abe45e-1cb1-44a2-b66e-a1e3520994ac-kube-api-access-bnzbt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.111158 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.111008    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01abe45e-1cb1-44a2-b66e-a1e3520994ac-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.111158 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.111055    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01abe45e-1cb1-44a2-b66e-a1e3520994ac-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.111158 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.111124    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01abe45e-1cb1-44a2-b66e-a1e3520994ac-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.111268 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.111172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01abe45e-1cb1-44a2-b66e-a1e3520994ac-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.111268 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.111204    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01abe45e-1cb1-44a2-b66e-a1e3520994ac-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.212097 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.212068    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01abe45e-1cb1-44a2-b66e-a1e3520994ac-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.212280 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.212102    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01abe45e-1cb1-44a2-b66e-a1e3520994ac-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.212280 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.212136    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01abe45e-1cb1-44a2-b66e-a1e3520994ac-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.212280 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.212160    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnzbt\" (UniqueName: \"kubernetes.io/projected/01abe45e-1cb1-44a2-b66e-a1e3520994ac-kube-api-access-bnzbt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.212280 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.212191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01abe45e-1cb1-44a2-b66e-a1e3520994ac-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.212280 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.212227    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01abe45e-1cb1-44a2-b66e-a1e3520994ac-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.212595 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.212576    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01abe45e-1cb1-44a2-b66e-a1e3520994ac-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.212645 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.212623    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01abe45e-1cb1-44a2-b66e-a1e3520994ac-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.212680 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.212664    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01abe45e-1cb1-44a2-b66e-a1e3520994ac-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.214583 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.214556    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01abe45e-1cb1-44a2-b66e-a1e3520994ac-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.214867 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.214849    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01abe45e-1cb1-44a2-b66e-a1e3520994ac-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.224727 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.224704    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnzbt\" (UniqueName: \"kubernetes.io/projected/01abe45e-1cb1-44a2-b66e-a1e3520994ac-kube-api-access-bnzbt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd\" (UID: \"01abe45e-1cb1-44a2-b66e-a1e3520994ac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.321593 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.321561    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:11.597979 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.597942    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"]
Sep 16 12:17:11.598454 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:17:11.598425    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01abe45e_1cb1_44a2_b66e_a1e3520994ac.slice/crio-499df3e88c49f148709de822fc4f16c0b467c09dc303f36745998469d8082d02 WatchSource:0}: Error finding container 499df3e88c49f148709de822fc4f16c0b467c09dc303f36745998469d8082d02: Status 404 returned error can't find the container with id 499df3e88c49f148709de822fc4f16c0b467c09dc303f36745998469d8082d02
Sep 16 12:17:11.599910 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:11.599885    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:17:12.337548 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:12.337509    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd" event={"ID":"01abe45e-1cb1-44a2-b66e-a1e3520994ac","Type":"ContainerStarted","Data":"499df3e88c49f148709de822fc4f16c0b467c09dc303f36745998469d8082d02"}
Sep 16 12:17:14.350821 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:14.350784    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd" event={"ID":"01abe45e-1cb1-44a2-b66e-a1e3520994ac","Type":"ContainerStarted","Data":"8ea07abf1e36d0c45f820d2958d6f495cebb7bfbf4d604bf2faaf1aa31dbddf5"}
Sep 16 12:17:14.371182 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:14.371129    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd" podStartSLOduration=2.261543729 podStartE2EDuration="4.371114205s" totalImagesPullingTime="2.109570476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:17:14.368852813 +0000 UTC m=+993.619390362" watchObservedRunningTime="2026-09-16 12:17:14.371114205 +0000 UTC m=+993.621651754"
Sep 16 12:17:21.322490 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:21.322443    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:21.323151 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:21.322533    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:21.335151 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:21.335122    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:21.395580 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:21.395552    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd"
Sep 16 12:17:33.903934 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:33.903889    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"]
Sep 16 12:17:33.907391 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:33.907372    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:33.910088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:33.910062    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 16 12:17:33.916434 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:33.916405    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"]
Sep 16 12:17:34.004010 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.003955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b29d12d-8575-43f5-8d11-15e5e85acdfc-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.004239 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.004117    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b29d12d-8575-43f5-8d11-15e5e85acdfc-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.004239 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.004166    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b29d12d-8575-43f5-8d11-15e5e85acdfc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.004239 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.004198    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsx4b\" (UniqueName: \"kubernetes.io/projected/6b29d12d-8575-43f5-8d11-15e5e85acdfc-kube-api-access-rsx4b\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.004239 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.004233    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b29d12d-8575-43f5-8d11-15e5e85acdfc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.004427 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.004289    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b29d12d-8575-43f5-8d11-15e5e85acdfc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.105584 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.105531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b29d12d-8575-43f5-8d11-15e5e85acdfc-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.105799 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.105635    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b29d12d-8575-43f5-8d11-15e5e85acdfc-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.105799 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.105665    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b29d12d-8575-43f5-8d11-15e5e85acdfc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.105799 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.105682    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsx4b\" (UniqueName: \"kubernetes.io/projected/6b29d12d-8575-43f5-8d11-15e5e85acdfc-kube-api-access-rsx4b\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.105799 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.105714    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b29d12d-8575-43f5-8d11-15e5e85acdfc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.105799 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.105778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b29d12d-8575-43f5-8d11-15e5e85acdfc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.106144 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.106121    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b29d12d-8575-43f5-8d11-15e5e85acdfc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.106188 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.106166    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b29d12d-8575-43f5-8d11-15e5e85acdfc-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.106227 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.106209    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b29d12d-8575-43f5-8d11-15e5e85acdfc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.108132 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.108103    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b29d12d-8575-43f5-8d11-15e5e85acdfc-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.108392 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.108375    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b29d12d-8575-43f5-8d11-15e5e85acdfc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.113662 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.113642    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsx4b\" (UniqueName: \"kubernetes.io/projected/6b29d12d-8575-43f5-8d11-15e5e85acdfc-kube-api-access-rsx4b\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f\" (UID: \"6b29d12d-8575-43f5-8d11-15e5e85acdfc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.217104 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.217009    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:34.487353 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:34.487324    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"]
Sep 16 12:17:34.489398 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:17:34.489368    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b29d12d_8575_43f5_8d11_15e5e85acdfc.slice/crio-5ea61b239a43b359398f6b18f83dab6303d05e20ed85ac543201ce13572fe3c6 WatchSource:0}: Error finding container 5ea61b239a43b359398f6b18f83dab6303d05e20ed85ac543201ce13572fe3c6: Status 404 returned error can't find the container with id 5ea61b239a43b359398f6b18f83dab6303d05e20ed85ac543201ce13572fe3c6
Sep 16 12:17:35.457116 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:35.457079    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f" event={"ID":"6b29d12d-8575-43f5-8d11-15e5e85acdfc","Type":"ContainerStarted","Data":"6f88996ef080778fb582480dc05da77e99efe0ec0c34659376b9470b5e4b940d"}
Sep 16 12:17:35.457539 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:35.457128    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f" event={"ID":"6b29d12d-8575-43f5-8d11-15e5e85acdfc","Type":"ContainerStarted","Data":"5ea61b239a43b359398f6b18f83dab6303d05e20ed85ac543201ce13572fe3c6"}
Sep 16 12:17:35.473791 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:35.473717    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f" podStartSLOduration=2.235877842 podStartE2EDuration="2.473702297s" totalImagesPullingTime="237.824455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:17:35.473012646 +0000 UTC m=+1014.723550194" watchObservedRunningTime="2026-09-16 12:17:35.473702297 +0000 UTC m=+1014.724239846"
Sep 16 12:17:44.217709 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:44.217666    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:44.218242 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:44.217759    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:44.231148 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:44.231106    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:17:44.519700 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:17:44.519606    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f"
Sep 16 12:18:57.541218 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:18:57.541137    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c677858cf-vqbww"]
Sep 16 12:18:57.544670 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:18:57.544646    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c677858cf-vqbww"
Sep 16 12:18:57.553170 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:18:57.553145    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c677858cf-vqbww"]
Sep 16 12:18:57.632464 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:18:57.632423    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmgjd\" (UniqueName: \"kubernetes.io/projected/fb8c8c07-9f8b-4f8c-9ceb-113c9a685e7e-kube-api-access-nmgjd\") pod \"authorino-6c677858cf-vqbww\" (UID: \"fb8c8c07-9f8b-4f8c-9ceb-113c9a685e7e\") " pod="kuadrant-system/authorino-6c677858cf-vqbww"
Sep 16 12:18:57.632655 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:18:57.632487    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb8c8c07-9f8b-4f8c-9ceb-113c9a685e7e-tls-cert\") pod \"authorino-6c677858cf-vqbww\" (UID: \"fb8c8c07-9f8b-4f8c-9ceb-113c9a685e7e\") " pod="kuadrant-system/authorino-6c677858cf-vqbww"
Sep 16 12:18:57.733690 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:18:57.733644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb8c8c07-9f8b-4f8c-9ceb-113c9a685e7e-tls-cert\") pod \"authorino-6c677858cf-vqbww\" (UID: \"fb8c8c07-9f8b-4f8c-9ceb-113c9a685e7e\") " pod="kuadrant-system/authorino-6c677858cf-vqbww"
Sep 16 12:18:57.733913 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:18:57.733809    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmgjd\" (UniqueName: \"kubernetes.io/projected/fb8c8c07-9f8b-4f8c-9ceb-113c9a685e7e-kube-api-access-nmgjd\") pod \"authorino-6c677858cf-vqbww\" (UID: \"fb8c8c07-9f8b-4f8c-9ceb-113c9a685e7e\") " pod="kuadrant-system/authorino-6c677858cf-vqbww"
Sep 16 12:18:57.736465 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:18:57.736429    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb8c8c07-9f8b-4f8c-9ceb-113c9a685e7e-tls-cert\") pod \"authorino-6c677858cf-vqbww\" (UID: \"fb8c8c07-9f8b-4f8c-9ceb-113c9a685e7e\") " pod="kuadrant-system/authorino-6c677858cf-vqbww"
Sep 16 12:18:57.742356 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:18:57.742331    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmgjd\" (UniqueName: \"kubernetes.io/projected/fb8c8c07-9f8b-4f8c-9ceb-113c9a685e7e-kube-api-access-nmgjd\") pod \"authorino-6c677858cf-vqbww\" (UID: \"fb8c8c07-9f8b-4f8c-9ceb-113c9a685e7e\") " pod="kuadrant-system/authorino-6c677858cf-vqbww"
Sep 16 12:18:57.854405 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:18:57.854350    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c677858cf-vqbww"
Sep 16 12:18:58.123973 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:18:58.123941    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c677858cf-vqbww"]
Sep 16 12:18:58.125222 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:18:58.125186    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb8c8c07_9f8b_4f8c_9ceb_113c9a685e7e.slice/crio-359eb270917e1fc1015c9a4a46dfc9df07e42e374901f958cb16794e37b015e8 WatchSource:0}: Error finding container 359eb270917e1fc1015c9a4a46dfc9df07e42e374901f958cb16794e37b015e8: Status 404 returned error can't find the container with id 359eb270917e1fc1015c9a4a46dfc9df07e42e374901f958cb16794e37b015e8
Sep 16 12:18:58.896616 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:18:58.896583    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c677858cf-vqbww" event={"ID":"fb8c8c07-9f8b-4f8c-9ceb-113c9a685e7e","Type":"ContainerStarted","Data":"359eb270917e1fc1015c9a4a46dfc9df07e42e374901f958cb16794e37b015e8"}
Sep 16 12:18:59.906829 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:18:59.906779    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c677858cf-vqbww" event={"ID":"fb8c8c07-9f8b-4f8c-9ceb-113c9a685e7e","Type":"ContainerStarted","Data":"44d9852ea4428cc91fb0506d51b9994cbad23a7fcb0458883a59bda6503bc484"}
Sep 16 12:18:59.926452 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:18:59.926383    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c677858cf-vqbww" podStartSLOduration=2.223723384 podStartE2EDuration="2.926361332s" totalImagesPullingTime="702.637948ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:18:59.92510631 +0000 UTC m=+1099.175643860" watchObservedRunningTime="2026-09-16 12:18:59.926361332 +0000 UTC m=+1099.176898883"
Sep 16 12:18:59.951057 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:18:59.951021    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79f8797474-zndjv"]
Sep 16 12:18:59.951283 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:18:59.951248    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-79f8797474-zndjv" podUID="964f5922-be1c-4f28-b064-976884195de2" containerName="authorino" containerID="cri-o://0b4198b88db99b6a3df7ded5e6bca0ffec670e7e4a8828ce9664aff86f43f64a" gracePeriod=30
Sep 16 12:19:00.202202 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:00.202174    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79f8797474-zndjv"
Sep 16 12:19:00.256861 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:00.256819    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/964f5922-be1c-4f28-b064-976884195de2-tls-cert\") pod \"964f5922-be1c-4f28-b064-976884195de2\" (UID: \"964f5922-be1c-4f28-b064-976884195de2\") "
Sep 16 12:19:00.257034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:00.256955    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml7lq\" (UniqueName: \"kubernetes.io/projected/964f5922-be1c-4f28-b064-976884195de2-kube-api-access-ml7lq\") pod \"964f5922-be1c-4f28-b064-976884195de2\" (UID: \"964f5922-be1c-4f28-b064-976884195de2\") "
Sep 16 12:19:00.259159 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:00.259124    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964f5922-be1c-4f28-b064-976884195de2-kube-api-access-ml7lq" (OuterVolumeSpecName: "kube-api-access-ml7lq") pod "964f5922-be1c-4f28-b064-976884195de2" (UID: "964f5922-be1c-4f28-b064-976884195de2"). InnerVolumeSpecName "kube-api-access-ml7lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:19:00.267490 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:00.267463    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/964f5922-be1c-4f28-b064-976884195de2-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "964f5922-be1c-4f28-b064-976884195de2" (UID: "964f5922-be1c-4f28-b064-976884195de2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:19:00.357819 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:00.357785    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml7lq\" (UniqueName: \"kubernetes.io/projected/964f5922-be1c-4f28-b064-976884195de2-kube-api-access-ml7lq\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:19:00.357819 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:00.357818    2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/964f5922-be1c-4f28-b064-976884195de2-tls-cert\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:19:00.920641 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:00.920593    2582 generic.go:356] "Generic (PLEG): container finished" podID="964f5922-be1c-4f28-b064-976884195de2" containerID="0b4198b88db99b6a3df7ded5e6bca0ffec670e7e4a8828ce9664aff86f43f64a" exitCode=0
Sep 16 12:19:00.921151 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:00.920703    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79f8797474-zndjv"
Sep 16 12:19:00.921151 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:00.920758    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79f8797474-zndjv" event={"ID":"964f5922-be1c-4f28-b064-976884195de2","Type":"ContainerDied","Data":"0b4198b88db99b6a3df7ded5e6bca0ffec670e7e4a8828ce9664aff86f43f64a"}
Sep 16 12:19:00.921151 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:00.920803    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79f8797474-zndjv" event={"ID":"964f5922-be1c-4f28-b064-976884195de2","Type":"ContainerDied","Data":"f6aa74ce40c4eac849b27ed9965605c0b946a57be65a22531e41608150e1832c"}
Sep 16 12:19:00.921151 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:00.920825    2582 scope.go:117] "RemoveContainer" containerID="0b4198b88db99b6a3df7ded5e6bca0ffec670e7e4a8828ce9664aff86f43f64a"
Sep 16 12:19:00.930772 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:00.930717    2582 scope.go:117] "RemoveContainer" containerID="0b4198b88db99b6a3df7ded5e6bca0ffec670e7e4a8828ce9664aff86f43f64a"
Sep 16 12:19:00.931139 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:19:00.931111    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4198b88db99b6a3df7ded5e6bca0ffec670e7e4a8828ce9664aff86f43f64a\": container with ID starting with 0b4198b88db99b6a3df7ded5e6bca0ffec670e7e4a8828ce9664aff86f43f64a not found: ID does not exist" containerID="0b4198b88db99b6a3df7ded5e6bca0ffec670e7e4a8828ce9664aff86f43f64a"
Sep 16 12:19:00.931244 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:00.931149    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4198b88db99b6a3df7ded5e6bca0ffec670e7e4a8828ce9664aff86f43f64a"} err="failed to get container status \"0b4198b88db99b6a3df7ded5e6bca0ffec670e7e4a8828ce9664aff86f43f64a\": rpc error: code = NotFound desc = could not find container \"0b4198b88db99b6a3df7ded5e6bca0ffec670e7e4a8828ce9664aff86f43f64a\": container with ID starting with 0b4198b88db99b6a3df7ded5e6bca0ffec670e7e4a8828ce9664aff86f43f64a not found: ID does not exist"
Sep 16 12:19:00.947269 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:00.947238    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79f8797474-zndjv"]
Sep 16 12:19:00.971029 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:00.970990    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-79f8797474-zndjv"]
Sep 16 12:19:01.212817 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:01.212694    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="964f5922-be1c-4f28-b064-976884195de2" path="/var/lib/kubelet/pods/964f5922-be1c-4f28-b064-976884195de2/volumes"
Sep 16 12:19:47.304309 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:47.304268    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7576-fb30c2"]
Sep 16 12:19:47.304951 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:47.304832    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="964f5922-be1c-4f28-b064-976884195de2" containerName="authorino"
Sep 16 12:19:47.304951 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:47.304854    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="964f5922-be1c-4f28-b064-976884195de2" containerName="authorino"
Sep 16 12:19:47.304951 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:47.304937    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="964f5922-be1c-4f28-b064-976884195de2" containerName="authorino"
Sep 16 12:19:47.309190 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:47.309166    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-fb30c2"
Sep 16 12:19:47.311868 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:47.311840    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7576-fb30c2"]
Sep 16 12:19:47.311868 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:47.311865    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h8vfr\""
Sep 16 12:19:47.359373 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:47.359340    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d2gt\" (UniqueName: \"kubernetes.io/projected/0f95918f-6ddc-418e-9ece-2f5a50e6e8b2-kube-api-access-8d2gt\") pod \"test-curl-7576-fb30c2\" (UID: \"0f95918f-6ddc-418e-9ece-2f5a50e6e8b2\") " pod="opendatahub/test-curl-7576-fb30c2"
Sep 16 12:19:47.460404 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:47.460371    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d2gt\" (UniqueName: \"kubernetes.io/projected/0f95918f-6ddc-418e-9ece-2f5a50e6e8b2-kube-api-access-8d2gt\") pod \"test-curl-7576-fb30c2\" (UID: \"0f95918f-6ddc-418e-9ece-2f5a50e6e8b2\") " pod="opendatahub/test-curl-7576-fb30c2"
Sep 16 12:19:47.468898 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:47.468875    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d2gt\" (UniqueName: \"kubernetes.io/projected/0f95918f-6ddc-418e-9ece-2f5a50e6e8b2-kube-api-access-8d2gt\") pod \"test-curl-7576-fb30c2\" (UID: \"0f95918f-6ddc-418e-9ece-2f5a50e6e8b2\") " pod="opendatahub/test-curl-7576-fb30c2"
Sep 16 12:19:47.619629 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:47.619595    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-fb30c2"
Sep 16 12:19:47.891817 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:47.891781    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7576-fb30c2"]
Sep 16 12:19:47.893566 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:19:47.893539    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f95918f_6ddc_418e_9ece_2f5a50e6e8b2.slice/crio-7991f4709cad1679c10a30c50ff290f92217c1bc9a176f62d52b9089cb30a688 WatchSource:0}: Error finding container 7991f4709cad1679c10a30c50ff290f92217c1bc9a176f62d52b9089cb30a688: Status 404 returned error can't find the container with id 7991f4709cad1679c10a30c50ff290f92217c1bc9a176f62d52b9089cb30a688
Sep 16 12:19:48.167763 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:48.167644    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7576-fb30c2" event={"ID":"0f95918f-6ddc-418e-9ece-2f5a50e6e8b2","Type":"ContainerStarted","Data":"7991f4709cad1679c10a30c50ff290f92217c1bc9a176f62d52b9089cb30a688"}
Sep 16 12:19:49.603946 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:49.603905    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"]
Sep 16 12:19:49.614865 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:49.614830    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:19:49.618440 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:49.617546    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-7a4d5c-serving-cert\""
Sep 16 12:19:49.618440 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:49.618218    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"]
Sep 16 12:19:49.683339 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:49.683295    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a048b31-681c-4da8-bdfd-7fe0d837fd67-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849\" (UID: \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:19:49.683529 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:49.683444    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vdtc\" (UniqueName: \"kubernetes.io/projected/4a048b31-681c-4da8-bdfd-7fe0d837fd67-kube-api-access-2vdtc\") pod \"maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849\" (UID: \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:19:49.683529 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:49.683504    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a048b31-681c-4da8-bdfd-7fe0d837fd67-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849\" (UID: \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:19:49.683652 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:49.683533    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a048b31-681c-4da8-bdfd-7fe0d837fd67-metrics-certs\") pod \"maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849\" (UID: \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:19:49.784837 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:49.784800    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a048b31-681c-4da8-bdfd-7fe0d837fd67-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849\" (UID: \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:19:49.785026 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:49.784875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vdtc\" (UniqueName: \"kubernetes.io/projected/4a048b31-681c-4da8-bdfd-7fe0d837fd67-kube-api-access-2vdtc\") pod \"maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849\" (UID: \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:19:49.785026 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:49.784907    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a048b31-681c-4da8-bdfd-7fe0d837fd67-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849\" (UID: \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:19:49.785026 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:49.784922    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a048b31-681c-4da8-bdfd-7fe0d837fd67-metrics-certs\") pod \"maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849\" (UID: \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:19:49.785026 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:19:49.784969    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-serving-cert: secret "maas-api-e2e-worker-w4-7a4d5c-serving-cert" not found
Sep 16 12:19:49.785207 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:19:49.785063    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a048b31-681c-4da8-bdfd-7fe0d837fd67-maas-api-tls podName:4a048b31-681c-4da8-bdfd-7fe0d837fd67 nodeName:}" failed. No retries permitted until 2026-09-16 12:19:50.285042161 +0000 UTC m=+1149.535579687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a048b31-681c-4da8-bdfd-7fe0d837fd67-maas-api-tls") pod "maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849" (UID: "4a048b31-681c-4da8-bdfd-7fe0d837fd67") : secret "maas-api-e2e-worker-w4-7a4d5c-serving-cert" not found
Sep 16 12:19:49.785564 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:49.785544    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a048b31-681c-4da8-bdfd-7fe0d837fd67-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849\" (UID: \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:19:49.787590 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:49.787567    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a048b31-681c-4da8-bdfd-7fe0d837fd67-metrics-certs\") pod \"maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849\" (UID: \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:19:49.794529 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:49.794507    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vdtc\" (UniqueName: \"kubernetes.io/projected/4a048b31-681c-4da8-bdfd-7fe0d837fd67-kube-api-access-2vdtc\") pod \"maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849\" (UID: \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:19:50.180958 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:50.180865    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7576-fb30c2" event={"ID":"0f95918f-6ddc-418e-9ece-2f5a50e6e8b2","Type":"ContainerStarted","Data":"31d167dfc0ff1ee81dcd129b0f1105c0eff5c6c78d33f47f4f5aa342972712a8"}
Sep 16 12:19:50.200806 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:50.200749    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7576-fb30c2" podStartSLOduration=1.233829416 podStartE2EDuration="3.200717721s" totalImagesPullingTime="1.966888305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:19:50.199204963 +0000 UTC m=+1149.449742544" watchObservedRunningTime="2026-09-16 12:19:50.200717721 +0000 UTC m=+1149.451255270"
Sep 16 12:19:50.289714 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:50.289673    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a048b31-681c-4da8-bdfd-7fe0d837fd67-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849\" (UID: \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:19:50.292557 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:50.292524    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a048b31-681c-4da8-bdfd-7fe0d837fd67-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849\" (UID: \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:19:50.529928 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:50.529840    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:19:50.790035 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:50.789947    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7576-fb30c2"]
Sep 16 12:19:50.794423 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:50.794381    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7576-fb30c2"]
Sep 16 12:19:50.804433 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:50.804409    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"]
Sep 16 12:19:50.808131 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:19:50.808100    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a048b31_681c_4da8_bdfd_7fe0d837fd67.slice/crio-a5c352f1dfa030cb663f629b5391b3cb9ad8ea2ae5559c60351d19d62e69cbea WatchSource:0}: Error finding container a5c352f1dfa030cb663f629b5391b3cb9ad8ea2ae5559c60351d19d62e69cbea: Status 404 returned error can't find the container with id a5c352f1dfa030cb663f629b5391b3cb9ad8ea2ae5559c60351d19d62e69cbea
Sep 16 12:19:50.914420 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:50.914384    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7576-52c257"]
Sep 16 12:19:50.914772 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:50.914758    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f95918f-6ddc-418e-9ece-2f5a50e6e8b2" containerName="test-curl-7576-fb30c2"
Sep 16 12:19:50.914772 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:50.914774    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f95918f-6ddc-418e-9ece-2f5a50e6e8b2" containerName="test-curl-7576-fb30c2"
Sep 16 12:19:50.914866 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:50.914849    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f95918f-6ddc-418e-9ece-2f5a50e6e8b2" containerName="test-curl-7576-fb30c2"
Sep 16 12:19:50.919122 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:50.919102    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-52c257"
Sep 16 12:19:50.923126 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:50.923099    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7576-52c257"]
Sep 16 12:19:50.996700 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:50.996664    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64dq4\" (UniqueName: \"kubernetes.io/projected/8bcccdd1-7210-4caa-bd1b-1a1da64a339e-kube-api-access-64dq4\") pod \"test-curl-7576-52c257\" (UID: \"8bcccdd1-7210-4caa-bd1b-1a1da64a339e\") " pod="opendatahub/test-curl-7576-52c257"
Sep 16 12:19:51.098323 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.098283    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64dq4\" (UniqueName: \"kubernetes.io/projected/8bcccdd1-7210-4caa-bd1b-1a1da64a339e-kube-api-access-64dq4\") pod \"test-curl-7576-52c257\" (UID: \"8bcccdd1-7210-4caa-bd1b-1a1da64a339e\") " pod="opendatahub/test-curl-7576-52c257"
Sep 16 12:19:51.106837 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.106811    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64dq4\" (UniqueName: \"kubernetes.io/projected/8bcccdd1-7210-4caa-bd1b-1a1da64a339e-kube-api-access-64dq4\") pod \"test-curl-7576-52c257\" (UID: \"8bcccdd1-7210-4caa-bd1b-1a1da64a339e\") " pod="opendatahub/test-curl-7576-52c257"
Sep 16 12:19:51.187386 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.187348    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849" event={"ID":"4a048b31-681c-4da8-bdfd-7fe0d837fd67","Type":"ContainerStarted","Data":"a5c352f1dfa030cb663f629b5391b3cb9ad8ea2ae5559c60351d19d62e69cbea"}
Sep 16 12:19:51.187609 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:19:51.187592    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7576-fb30c2" container="test-curl-7576-fb30c2"
Sep 16 12:19:51.230787 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.230754    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-52c257"
Sep 16 12:19:51.286974 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.286936    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"]
Sep 16 12:19:51.292544 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.292519    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:19:51.296308 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.296270    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-0e8ffe-serving-cert\""
Sep 16 12:19:51.299794 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.299679    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"]
Sep 16 12:19:51.400501 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.400473    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c60d328f-e42b-46d0-b92c-dccf10b62790-metrics-certs\") pod \"maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v\" (UID: \"c60d328f-e42b-46d0-b92c-dccf10b62790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:19:51.400596 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.400529    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c60d328f-e42b-46d0-b92c-dccf10b62790-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v\" (UID: \"c60d328f-e42b-46d0-b92c-dccf10b62790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:19:51.400668 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.400644    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c60d328f-e42b-46d0-b92c-dccf10b62790-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v\" (UID: \"c60d328f-e42b-46d0-b92c-dccf10b62790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:19:51.400717 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.400704    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6xsf\" (UniqueName: \"kubernetes.io/projected/c60d328f-e42b-46d0-b92c-dccf10b62790-kube-api-access-z6xsf\") pod \"maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v\" (UID: \"c60d328f-e42b-46d0-b92c-dccf10b62790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:19:51.501613 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.501492    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6xsf\" (UniqueName: \"kubernetes.io/projected/c60d328f-e42b-46d0-b92c-dccf10b62790-kube-api-access-z6xsf\") pod \"maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v\" (UID: \"c60d328f-e42b-46d0-b92c-dccf10b62790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:19:51.501613 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.501600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c60d328f-e42b-46d0-b92c-dccf10b62790-metrics-certs\") pod \"maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v\" (UID: \"c60d328f-e42b-46d0-b92c-dccf10b62790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:19:51.501807 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.501643    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c60d328f-e42b-46d0-b92c-dccf10b62790-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v\" (UID: \"c60d328f-e42b-46d0-b92c-dccf10b62790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:19:51.501807 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.501681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c60d328f-e42b-46d0-b92c-dccf10b62790-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v\" (UID: \"c60d328f-e42b-46d0-b92c-dccf10b62790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:19:51.502428 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.502398    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c60d328f-e42b-46d0-b92c-dccf10b62790-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v\" (UID: \"c60d328f-e42b-46d0-b92c-dccf10b62790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:19:51.504665 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.504633    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7576-52c257"]
Sep 16 12:19:51.504809 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.504790    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c60d328f-e42b-46d0-b92c-dccf10b62790-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v\" (UID: \"c60d328f-e42b-46d0-b92c-dccf10b62790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:19:51.504869 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.504818    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c60d328f-e42b-46d0-b92c-dccf10b62790-metrics-certs\") pod \"maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v\" (UID: \"c60d328f-e42b-46d0-b92c-dccf10b62790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:19:51.505419 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:19:51.505397    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bcccdd1_7210_4caa_bd1b_1a1da64a339e.slice/crio-07d402d3f546cd9e4c59ee9eb2041492e39e26686193dd79d5a98f89514f5533 WatchSource:0}: Error finding container 07d402d3f546cd9e4c59ee9eb2041492e39e26686193dd79d5a98f89514f5533: Status 404 returned error can't find the container with id 07d402d3f546cd9e4c59ee9eb2041492e39e26686193dd79d5a98f89514f5533
Sep 16 12:19:51.515601 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.515575    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6xsf\" (UniqueName: \"kubernetes.io/projected/c60d328f-e42b-46d0-b92c-dccf10b62790-kube-api-access-z6xsf\") pod \"maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v\" (UID: \"c60d328f-e42b-46d0-b92c-dccf10b62790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:19:51.609308 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.609276    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:19:51.899399 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:51.899366    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"]
Sep 16 12:19:51.901660 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:19:51.901627    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60d328f_e42b_46d0_b92c_dccf10b62790.slice/crio-bc864b5955ebd3bbcea17209f082f98a0c296dc03f347d6bb49099ca4e5fcba9 WatchSource:0}: Error finding container bc864b5955ebd3bbcea17209f082f98a0c296dc03f347d6bb49099ca4e5fcba9: Status 404 returned error can't find the container with id bc864b5955ebd3bbcea17209f082f98a0c296dc03f347d6bb49099ca4e5fcba9
Sep 16 12:19:52.194362 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:52.194260    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7576-52c257" event={"ID":"8bcccdd1-7210-4caa-bd1b-1a1da64a339e","Type":"ContainerStarted","Data":"ee942e41119bc17fc07478d72175d48ef231f223cc0eed511b6fc213d4fd49a0"}
Sep 16 12:19:52.194362 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:52.194307    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7576-52c257" event={"ID":"8bcccdd1-7210-4caa-bd1b-1a1da64a339e","Type":"ContainerStarted","Data":"07d402d3f546cd9e4c59ee9eb2041492e39e26686193dd79d5a98f89514f5533"}
Sep 16 12:19:52.196048 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:52.196006    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849" event={"ID":"4a048b31-681c-4da8-bdfd-7fe0d837fd67","Type":"ContainerStarted","Data":"997c8555c0db6c0beef10df5e2d0e7bad12b9a1db3bfcdc6f0cee42652f5beab"}
Sep 16 12:19:52.196194 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:52.196076    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:19:52.197263 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:52.197239    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v" event={"ID":"c60d328f-e42b-46d0-b92c-dccf10b62790","Type":"ContainerStarted","Data":"bc864b5955ebd3bbcea17209f082f98a0c296dc03f347d6bb49099ca4e5fcba9"}
Sep 16 12:19:52.197375 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:52.197286    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7576-fb30c2" podUID="0f95918f-6ddc-418e-9ece-2f5a50e6e8b2" containerName="test-curl-7576-fb30c2" containerID="cri-o://31d167dfc0ff1ee81dcd129b0f1105c0eff5c6c78d33f47f4f5aa342972712a8" gracePeriod=2
Sep 16 12:19:52.231039 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:52.230979    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7576-52c257" podStartSLOduration=2.230963697 podStartE2EDuration="2.230963697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:19:52.228306186 +0000 UTC m=+1151.478843735" watchObservedRunningTime="2026-09-16 12:19:52.230963697 +0000 UTC m=+1151.481501253"
Sep 16 12:19:52.277638 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:52.277588    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849" podStartSLOduration=2.7176003619999998 podStartE2EDuration="3.277571116s" totalImagesPullingTime="559.970754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:19:52.277296112 +0000 UTC m=+1151.527833662" watchObservedRunningTime="2026-09-16 12:19:52.277571116 +0000 UTC m=+1151.528108697"
Sep 16 12:19:52.826518 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:52.826485    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7576-52c257"]
Sep 16 12:19:52.830953 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:52.830923    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7576-52c257"]
Sep 16 12:19:53.204422 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.204380    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v" event={"ID":"c60d328f-e42b-46d0-b92c-dccf10b62790","Type":"ContainerStarted","Data":"bfca2629f843d7443b68b11637dfa256f32dd4222718d952d4eb84730781ed2c"}
Sep 16 12:19:53.204905 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.204551    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:19:53.204905 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:19:53.204608    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7576-52c257" container="test-curl-7576-52c257"
Sep 16 12:19:53.233333 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.233279    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v" podStartSLOduration=1.867532249 podStartE2EDuration="2.233260754s" totalImagesPullingTime="365.728505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:19:53.230985213 +0000 UTC m=+1152.481522763" watchObservedRunningTime="2026-09-16 12:19:53.233260754 +0000 UTC m=+1152.483798338"
Sep 16 12:19:53.495271 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.495182    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"]
Sep 16 12:19:53.495705 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.495683    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bcccdd1-7210-4caa-bd1b-1a1da64a339e" containerName="test-curl-7576-52c257"
Sep 16 12:19:53.495705 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.495707    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bcccdd1-7210-4caa-bd1b-1a1da64a339e" containerName="test-curl-7576-52c257"
Sep 16 12:19:53.495906 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.495856    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bcccdd1-7210-4caa-bd1b-1a1da64a339e" containerName="test-curl-7576-52c257"
Sep 16 12:19:53.499264 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.499242    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:19:53.501841 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.501800    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9a49af-serving-cert\""
Sep 16 12:19:53.510046 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.510016    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"]
Sep 16 12:19:53.621432 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.621393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e408e5-9ee5-4fcb-ba46-64f031e812f2-metrics-certs\") pod \"maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s\" (UID: \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:19:53.621611 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.621438    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e408e5-9ee5-4fcb-ba46-64f031e812f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s\" (UID: \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:19:53.621611 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.621470    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e408e5-9ee5-4fcb-ba46-64f031e812f2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s\" (UID: \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:19:53.621611 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.621560    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxbk\" (UniqueName: \"kubernetes.io/projected/33e408e5-9ee5-4fcb-ba46-64f031e812f2-kube-api-access-tlxbk\") pod \"maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s\" (UID: \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:19:53.722861 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.722825    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxbk\" (UniqueName: \"kubernetes.io/projected/33e408e5-9ee5-4fcb-ba46-64f031e812f2-kube-api-access-tlxbk\") pod \"maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s\" (UID: \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:19:53.723056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.722885    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e408e5-9ee5-4fcb-ba46-64f031e812f2-metrics-certs\") pod \"maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s\" (UID: \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:19:53.723056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.722911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e408e5-9ee5-4fcb-ba46-64f031e812f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s\" (UID: \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:19:53.723056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.722933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e408e5-9ee5-4fcb-ba46-64f031e812f2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s\" (UID: \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:19:53.723056 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:19:53.723048    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-serving-cert: secret "maas-api-e2e-inf-a-9a49af-serving-cert" not found
Sep 16 12:19:53.723201 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:19:53.723126    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33e408e5-9ee5-4fcb-ba46-64f031e812f2-maas-api-tls podName:33e408e5-9ee5-4fcb-ba46-64f031e812f2 nodeName:}" failed. No retries permitted until 2026-09-16 12:19:54.22310525 +0000 UTC m=+1153.473642777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33e408e5-9ee5-4fcb-ba46-64f031e812f2-maas-api-tls") pod "maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s" (UID: "33e408e5-9ee5-4fcb-ba46-64f031e812f2") : secret "maas-api-e2e-inf-a-9a49af-serving-cert" not found
Sep 16 12:19:53.723550 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.723521    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e408e5-9ee5-4fcb-ba46-64f031e812f2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s\" (UID: \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:19:53.725537 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.725517    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e408e5-9ee5-4fcb-ba46-64f031e812f2-metrics-certs\") pod \"maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s\" (UID: \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:19:53.736593 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:53.736569    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxbk\" (UniqueName: \"kubernetes.io/projected/33e408e5-9ee5-4fcb-ba46-64f031e812f2-kube-api-access-tlxbk\") pod \"maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s\" (UID: \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:19:54.209902 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:54.209850    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7576-52c257" podUID="8bcccdd1-7210-4caa-bd1b-1a1da64a339e" containerName="test-curl-7576-52c257" containerID="cri-o://ee942e41119bc17fc07478d72175d48ef231f223cc0eed511b6fc213d4fd49a0" gracePeriod=2
Sep 16 12:19:54.227945 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:54.227910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e408e5-9ee5-4fcb-ba46-64f031e812f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s\" (UID: \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:19:54.230668 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:54.230641    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e408e5-9ee5-4fcb-ba46-64f031e812f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s\" (UID: \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:19:54.371467 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:54.371438    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-fb30c2"
Sep 16 12:19:54.410946 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:54.410909    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:19:54.430054 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:54.430018    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d2gt\" (UniqueName: \"kubernetes.io/projected/0f95918f-6ddc-418e-9ece-2f5a50e6e8b2-kube-api-access-8d2gt\") pod \"0f95918f-6ddc-418e-9ece-2f5a50e6e8b2\" (UID: \"0f95918f-6ddc-418e-9ece-2f5a50e6e8b2\") "
Sep 16 12:19:54.432417 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:54.432391    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f95918f-6ddc-418e-9ece-2f5a50e6e8b2-kube-api-access-8d2gt" (OuterVolumeSpecName: "kube-api-access-8d2gt") pod "0f95918f-6ddc-418e-9ece-2f5a50e6e8b2" (UID: "0f95918f-6ddc-418e-9ece-2f5a50e6e8b2"). InnerVolumeSpecName "kube-api-access-8d2gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:19:54.531041 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:54.531010    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d2gt\" (UniqueName: \"kubernetes.io/projected/0f95918f-6ddc-418e-9ece-2f5a50e6e8b2-kube-api-access-8d2gt\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:19:54.688021 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:54.687992    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"]
Sep 16 12:19:54.689637 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:19:54.689609    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e408e5_9ee5_4fcb_ba46_64f031e812f2.slice/crio-f9ba8f9348523e61c9ee1e685a7dbf1cff4260f5115be3f642c17fb485167ec7 WatchSource:0}: Error finding container f9ba8f9348523e61c9ee1e685a7dbf1cff4260f5115be3f642c17fb485167ec7: Status 404 returned error can't find the container with id f9ba8f9348523e61c9ee1e685a7dbf1cff4260f5115be3f642c17fb485167ec7
Sep 16 12:19:55.212807 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:55.212768    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f95918f-6ddc-418e-9ece-2f5a50e6e8b2" path="/var/lib/kubelet/pods/0f95918f-6ddc-418e-9ece-2f5a50e6e8b2/volumes"
Sep 16 12:19:55.216944 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:55.216915    2582 generic.go:356] "Generic (PLEG): container finished" podID="0f95918f-6ddc-418e-9ece-2f5a50e6e8b2" containerID="31d167dfc0ff1ee81dcd129b0f1105c0eff5c6c78d33f47f4f5aa342972712a8" exitCode=137
Sep 16 12:19:55.217126 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:55.216969    2582 scope.go:117] "RemoveContainer" containerID="31d167dfc0ff1ee81dcd129b0f1105c0eff5c6c78d33f47f4f5aa342972712a8"
Sep 16 12:19:55.217126 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:55.216990    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-fb30c2"
Sep 16 12:19:55.219049 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:55.219027    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s" event={"ID":"33e408e5-9ee5-4fcb-ba46-64f031e812f2","Type":"ContainerStarted","Data":"8a397118c4e5bdef9ae02f34cbaff1f9ef18486ad4d8a479c95e78466897515c"}
Sep 16 12:19:55.219173 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:55.219082    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s" event={"ID":"33e408e5-9ee5-4fcb-ba46-64f031e812f2","Type":"ContainerStarted","Data":"f9ba8f9348523e61c9ee1e685a7dbf1cff4260f5115be3f642c17fb485167ec7"}
Sep 16 12:19:55.219173 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:55.219111    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:19:55.229356 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:55.229299    2582 scope.go:117] "RemoveContainer" containerID="31d167dfc0ff1ee81dcd129b0f1105c0eff5c6c78d33f47f4f5aa342972712a8"
Sep 16 12:19:55.229861 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:19:55.229837    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d167dfc0ff1ee81dcd129b0f1105c0eff5c6c78d33f47f4f5aa342972712a8\": container with ID starting with 31d167dfc0ff1ee81dcd129b0f1105c0eff5c6c78d33f47f4f5aa342972712a8 not found: ID does not exist" containerID="31d167dfc0ff1ee81dcd129b0f1105c0eff5c6c78d33f47f4f5aa342972712a8"
Sep 16 12:19:55.229966 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:55.229871    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d167dfc0ff1ee81dcd129b0f1105c0eff5c6c78d33f47f4f5aa342972712a8"} err="failed to get container status \"31d167dfc0ff1ee81dcd129b0f1105c0eff5c6c78d33f47f4f5aa342972712a8\": rpc error: code = NotFound desc = could not find container \"31d167dfc0ff1ee81dcd129b0f1105c0eff5c6c78d33f47f4f5aa342972712a8\": container with ID starting with 31d167dfc0ff1ee81dcd129b0f1105c0eff5c6c78d33f47f4f5aa342972712a8 not found: ID does not exist"
Sep 16 12:19:55.266631 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:55.266557    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s" podStartSLOduration=1.853639588 podStartE2EDuration="2.266536696s" totalImagesPullingTime="412.897108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:19:55.26294972 +0000 UTC m=+1154.513487269" watchObservedRunningTime="2026-09-16 12:19:55.266536696 +0000 UTC m=+1154.517074247"
Sep 16 12:19:56.403169 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:56.403136    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-52c257"
Sep 16 12:19:56.549190 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:56.549101    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64dq4\" (UniqueName: \"kubernetes.io/projected/8bcccdd1-7210-4caa-bd1b-1a1da64a339e-kube-api-access-64dq4\") pod \"8bcccdd1-7210-4caa-bd1b-1a1da64a339e\" (UID: \"8bcccdd1-7210-4caa-bd1b-1a1da64a339e\") "
Sep 16 12:19:56.551614 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:56.551580    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bcccdd1-7210-4caa-bd1b-1a1da64a339e-kube-api-access-64dq4" (OuterVolumeSpecName: "kube-api-access-64dq4") pod "8bcccdd1-7210-4caa-bd1b-1a1da64a339e" (UID: "8bcccdd1-7210-4caa-bd1b-1a1da64a339e"). InnerVolumeSpecName "kube-api-access-64dq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:19:56.650049 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:56.650008    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64dq4\" (UniqueName: \"kubernetes.io/projected/8bcccdd1-7210-4caa-bd1b-1a1da64a339e-kube-api-access-64dq4\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:19:57.214540 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:57.214499    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bcccdd1-7210-4caa-bd1b-1a1da64a339e" path="/var/lib/kubelet/pods/8bcccdd1-7210-4caa-bd1b-1a1da64a339e/volumes"
Sep 16 12:19:57.235194 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:57.235156    2582 generic.go:356] "Generic (PLEG): container finished" podID="8bcccdd1-7210-4caa-bd1b-1a1da64a339e" containerID="ee942e41119bc17fc07478d72175d48ef231f223cc0eed511b6fc213d4fd49a0" exitCode=137
Sep 16 12:19:57.235352 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:57.235202    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-52c257"
Sep 16 12:19:57.235352 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:57.235238    2582 scope.go:117] "RemoveContainer" containerID="ee942e41119bc17fc07478d72175d48ef231f223cc0eed511b6fc213d4fd49a0"
Sep 16 12:19:57.244140 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:57.244115    2582 scope.go:117] "RemoveContainer" containerID="ee942e41119bc17fc07478d72175d48ef231f223cc0eed511b6fc213d4fd49a0"
Sep 16 12:19:57.244480 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:19:57.244459    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee942e41119bc17fc07478d72175d48ef231f223cc0eed511b6fc213d4fd49a0\": container with ID starting with ee942e41119bc17fc07478d72175d48ef231f223cc0eed511b6fc213d4fd49a0 not found: ID does not exist" containerID="ee942e41119bc17fc07478d72175d48ef231f223cc0eed511b6fc213d4fd49a0"
Sep 16 12:19:57.244551 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:57.244488    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee942e41119bc17fc07478d72175d48ef231f223cc0eed511b6fc213d4fd49a0"} err="failed to get container status \"ee942e41119bc17fc07478d72175d48ef231f223cc0eed511b6fc213d4fd49a0\": rpc error: code = NotFound desc = could not find container \"ee942e41119bc17fc07478d72175d48ef231f223cc0eed511b6fc213d4fd49a0\": container with ID starting with ee942e41119bc17fc07478d72175d48ef231f223cc0eed511b6fc213d4fd49a0 not found: ID does not exist"
Sep 16 12:19:58.210810 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:58.210775    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:19:59.215811 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:59.215777    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:19:59.894537 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:59.894501    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"]
Sep 16 12:19:59.899553 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:59.899526    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:19:59.902264 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:59.902240    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1ce21877-serving-cert\""
Sep 16 12:19:59.911211 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:59.911178    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"]
Sep 16 12:19:59.980294 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:59.980261    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkxlk\" (UniqueName: \"kubernetes.io/projected/7c61d966-b6b0-4602-8a20-289cc62dd512-kube-api-access-rkxlk\") pod \"maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk\" (UID: \"7c61d966-b6b0-4602-8a20-289cc62dd512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:19:59.980461 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:59.980314    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c61d966-b6b0-4602-8a20-289cc62dd512-maas-api-tls\") pod \"maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk\" (UID: \"7c61d966-b6b0-4602-8a20-289cc62dd512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:19:59.980461 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:59.980342    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c61d966-b6b0-4602-8a20-289cc62dd512-metrics-certs\") pod \"maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk\" (UID: \"7c61d966-b6b0-4602-8a20-289cc62dd512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:19:59.980461 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:19:59.980402    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c61d966-b6b0-4602-8a20-289cc62dd512-openshift-service-ca\") pod \"maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk\" (UID: \"7c61d966-b6b0-4602-8a20-289cc62dd512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:20:00.081817 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:00.081781    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c61d966-b6b0-4602-8a20-289cc62dd512-maas-api-tls\") pod \"maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk\" (UID: \"7c61d966-b6b0-4602-8a20-289cc62dd512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:20:00.081817 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:00.081819    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c61d966-b6b0-4602-8a20-289cc62dd512-metrics-certs\") pod \"maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk\" (UID: \"7c61d966-b6b0-4602-8a20-289cc62dd512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:20:00.082097 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:00.081841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c61d966-b6b0-4602-8a20-289cc62dd512-openshift-service-ca\") pod \"maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk\" (UID: \"7c61d966-b6b0-4602-8a20-289cc62dd512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:20:00.082097 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:00.081905    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkxlk\" (UniqueName: \"kubernetes.io/projected/7c61d966-b6b0-4602-8a20-289cc62dd512-kube-api-access-rkxlk\") pod \"maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk\" (UID: \"7c61d966-b6b0-4602-8a20-289cc62dd512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:20:00.082097 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:20:00.081946    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-serving-cert: secret "maas-api-e2e-ait-1ce21877-serving-cert" not found
Sep 16 12:20:00.082097 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:20:00.082024    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c61d966-b6b0-4602-8a20-289cc62dd512-maas-api-tls podName:7c61d966-b6b0-4602-8a20-289cc62dd512 nodeName:}" failed. No retries permitted until 2026-09-16 12:20:00.582004033 +0000 UTC m=+1159.832541560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c61d966-b6b0-4602-8a20-289cc62dd512-maas-api-tls") pod "maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk" (UID: "7c61d966-b6b0-4602-8a20-289cc62dd512") : secret "maas-api-e2e-ait-1ce21877-serving-cert" not found
Sep 16 12:20:00.082604 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:00.082586    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c61d966-b6b0-4602-8a20-289cc62dd512-openshift-service-ca\") pod \"maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk\" (UID: \"7c61d966-b6b0-4602-8a20-289cc62dd512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:20:00.084471 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:00.084442    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c61d966-b6b0-4602-8a20-289cc62dd512-metrics-certs\") pod \"maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk\" (UID: \"7c61d966-b6b0-4602-8a20-289cc62dd512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:20:00.090572 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:00.090546    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkxlk\" (UniqueName: \"kubernetes.io/projected/7c61d966-b6b0-4602-8a20-289cc62dd512-kube-api-access-rkxlk\") pod \"maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk\" (UID: \"7c61d966-b6b0-4602-8a20-289cc62dd512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:20:00.587141 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:00.587100    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c61d966-b6b0-4602-8a20-289cc62dd512-maas-api-tls\") pod \"maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk\" (UID: \"7c61d966-b6b0-4602-8a20-289cc62dd512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:20:00.589696 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:00.589661    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c61d966-b6b0-4602-8a20-289cc62dd512-maas-api-tls\") pod \"maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk\" (UID: \"7c61d966-b6b0-4602-8a20-289cc62dd512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:20:00.812396 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:00.812355    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:20:01.087061 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.087036    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"]
Sep 16 12:20:01.088274 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:20:01.088254    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c61d966_b6b0_4602_8a20_289cc62dd512.slice/crio-80a40312481562739d010236528917a592689012e8756ba7fb91e7ccd8566a54 WatchSource:0}: Error finding container 80a40312481562739d010236528917a592689012e8756ba7fb91e7ccd8566a54: Status 404 returned error can't find the container with id 80a40312481562739d010236528917a592689012e8756ba7fb91e7ccd8566a54
Sep 16 12:20:01.234293 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.234206    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:20:01.260633 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.260599    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk" event={"ID":"7c61d966-b6b0-4602-8a20-289cc62dd512","Type":"ContainerStarted","Data":"80a40312481562739d010236528917a592689012e8756ba7fb91e7ccd8566a54"}
Sep 16 12:20:01.694982 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.694943    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"]
Sep 16 12:20:01.699881 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.699853    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:20:01.702676 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.702645    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-7c3429-serving-cert\""
Sep 16 12:20:01.713582 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.713518    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"]
Sep 16 12:20:01.798207 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.798169    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s8hv\" (UniqueName: \"kubernetes.io/projected/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-kube-api-access-9s8hv\") pod \"maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2\" (UID: \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:20:01.798392 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.798239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-metrics-certs\") pod \"maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2\" (UID: \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:20:01.798392 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.798277    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-maas-api-tls\") pod \"maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2\" (UID: \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:20:01.798392 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.798302    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-openshift-service-ca\") pod \"maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2\" (UID: \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:20:01.899525 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.899497    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s8hv\" (UniqueName: \"kubernetes.io/projected/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-kube-api-access-9s8hv\") pod \"maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2\" (UID: \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:20:01.899648 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.899561    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-metrics-certs\") pod \"maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2\" (UID: \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:20:01.899648 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.899619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-maas-api-tls\") pod \"maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2\" (UID: \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:20:01.899808 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.899657    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-openshift-service-ca\") pod \"maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2\" (UID: \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:20:01.899808 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:20:01.899784    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-serving-cert: secret "maas-api-e2e-autores-7c3429-serving-cert" not found
Sep 16 12:20:01.899898 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:20:01.899876    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-maas-api-tls podName:0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4 nodeName:}" failed. No retries permitted until 2026-09-16 12:20:02.399854285 +0000 UTC m=+1161.650391835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-maas-api-tls") pod "maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2" (UID: "0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4") : secret "maas-api-e2e-autores-7c3429-serving-cert" not found
Sep 16 12:20:01.900438 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.900376    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-openshift-service-ca\") pod \"maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2\" (UID: \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:20:01.902181 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.902157    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-metrics-certs\") pod \"maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2\" (UID: \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:20:01.909162 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:01.909136    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s8hv\" (UniqueName: \"kubernetes.io/projected/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-kube-api-access-9s8hv\") pod \"maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2\" (UID: \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:20:02.268553 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:02.268455    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk" event={"ID":"7c61d966-b6b0-4602-8a20-289cc62dd512","Type":"ContainerStarted","Data":"4cb8ebf1416722dc2a62f2bb29ea87d02df8adc62ab5c888ac71eb26aa14cf08"}
Sep 16 12:20:02.268553 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:02.268545    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:20:02.292015 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:02.291958    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk" podStartSLOduration=2.531683497 podStartE2EDuration="3.291939007s" totalImagesPullingTime="760.25551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:02.290466008 +0000 UTC m=+1161.541003559" watchObservedRunningTime="2026-09-16 12:20:02.291939007 +0000 UTC m=+1161.542476553"
Sep 16 12:20:02.404219 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:02.404178    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-maas-api-tls\") pod \"maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2\" (UID: \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:20:02.407005 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:02.406982    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-maas-api-tls\") pod \"maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2\" (UID: \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:20:02.615045 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:02.615005    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:20:02.892428 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:02.892393    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"]
Sep 16 12:20:02.893706 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:20:02.893677    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3f3a7c_b235_4f34_83a1_c3e8b8fdcbe4.slice/crio-0a21ed4ff7dfb97acb4f4f3631ed0fd571a4db6fb029fd20ad44c89ce2914c07 WatchSource:0}: Error finding container 0a21ed4ff7dfb97acb4f4f3631ed0fd571a4db6fb029fd20ad44c89ce2914c07: Status 404 returned error can't find the container with id 0a21ed4ff7dfb97acb4f4f3631ed0fd571a4db6fb029fd20ad44c89ce2914c07
Sep 16 12:20:03.275645 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:03.275555    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2" event={"ID":"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4","Type":"ContainerStarted","Data":"0a21ed4ff7dfb97acb4f4f3631ed0fd571a4db6fb029fd20ad44c89ce2914c07"}
Sep 16 12:20:04.282755 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:04.282704    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2" event={"ID":"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4","Type":"ContainerStarted","Data":"92577b7f6ca1b38aa68e807f08ab87b933557dec1d0cf42920df1f3197764c35"}
Sep 16 12:20:04.283173 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:04.282788    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:20:04.306797 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:04.304618    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2" podStartSLOduration=2.7035088590000003 podStartE2EDuration="3.304594832s" totalImagesPullingTime="601.085973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:04.300221223 +0000 UTC m=+1163.550758772" watchObservedRunningTime="2026-09-16 12:20:04.304594832 +0000 UTC m=+1163.555132383"
Sep 16 12:20:08.281442 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:08.281414    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:20:10.293417 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:10.293389    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:20:15.429962 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:15.429922    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"]
Sep 16 12:20:15.433813 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:15.433787    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:20:15.436937 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:15.436914    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4bef4d-serving-cert\""
Sep 16 12:20:15.446827 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:15.446801    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"]
Sep 16 12:20:15.533843 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:15.533804    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d62ba3d-5219-4153-b13a-ec54a22e919a-maas-api-tls\") pod \"maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7\" (UID: \"3d62ba3d-5219-4153-b13a-ec54a22e919a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:20:15.533843 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:15.533843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d62ba3d-5219-4153-b13a-ec54a22e919a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7\" (UID: \"3d62ba3d-5219-4153-b13a-ec54a22e919a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:20:15.534091 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:15.533997    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d62ba3d-5219-4153-b13a-ec54a22e919a-metrics-certs\") pod \"maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7\" (UID: \"3d62ba3d-5219-4153-b13a-ec54a22e919a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:20:15.534091 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:15.534068    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-648lp\" (UniqueName: \"kubernetes.io/projected/3d62ba3d-5219-4153-b13a-ec54a22e919a-kube-api-access-648lp\") pod \"maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7\" (UID: \"3d62ba3d-5219-4153-b13a-ec54a22e919a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:20:15.634758 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:15.634696    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d62ba3d-5219-4153-b13a-ec54a22e919a-maas-api-tls\") pod \"maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7\" (UID: \"3d62ba3d-5219-4153-b13a-ec54a22e919a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:20:15.634949 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:15.634771    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d62ba3d-5219-4153-b13a-ec54a22e919a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7\" (UID: \"3d62ba3d-5219-4153-b13a-ec54a22e919a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:20:15.634949 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:20:15.634850    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-serving-cert: secret "maas-api-e2e-inf-b-4bef4d-serving-cert" not found
Sep 16 12:20:15.634949 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:15.634887    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d62ba3d-5219-4153-b13a-ec54a22e919a-metrics-certs\") pod \"maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7\" (UID: \"3d62ba3d-5219-4153-b13a-ec54a22e919a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:20:15.634949 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:20:15.634930    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d62ba3d-5219-4153-b13a-ec54a22e919a-maas-api-tls podName:3d62ba3d-5219-4153-b13a-ec54a22e919a nodeName:}" failed. No retries permitted until 2026-09-16 12:20:16.13490823 +0000 UTC m=+1175.385445776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d62ba3d-5219-4153-b13a-ec54a22e919a-maas-api-tls") pod "maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7" (UID: "3d62ba3d-5219-4153-b13a-ec54a22e919a") : secret "maas-api-e2e-inf-b-4bef4d-serving-cert" not found
Sep 16 12:20:15.635159 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:15.634978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-648lp\" (UniqueName: \"kubernetes.io/projected/3d62ba3d-5219-4153-b13a-ec54a22e919a-kube-api-access-648lp\") pod \"maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7\" (UID: \"3d62ba3d-5219-4153-b13a-ec54a22e919a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:20:15.635437 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:15.635416    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d62ba3d-5219-4153-b13a-ec54a22e919a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7\" (UID: \"3d62ba3d-5219-4153-b13a-ec54a22e919a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:20:15.637420 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:15.637401    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d62ba3d-5219-4153-b13a-ec54a22e919a-metrics-certs\") pod \"maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7\" (UID: \"3d62ba3d-5219-4153-b13a-ec54a22e919a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:20:15.649786 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:15.649762    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-648lp\" (UniqueName: \"kubernetes.io/projected/3d62ba3d-5219-4153-b13a-ec54a22e919a-kube-api-access-648lp\") pod \"maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7\" (UID: \"3d62ba3d-5219-4153-b13a-ec54a22e919a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:20:16.138962 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:16.138918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d62ba3d-5219-4153-b13a-ec54a22e919a-maas-api-tls\") pod \"maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7\" (UID: \"3d62ba3d-5219-4153-b13a-ec54a22e919a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:20:16.141538 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:16.141514    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d62ba3d-5219-4153-b13a-ec54a22e919a-maas-api-tls\") pod \"maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7\" (UID: \"3d62ba3d-5219-4153-b13a-ec54a22e919a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:20:16.346723 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:16.346685    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:20:16.620394 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:16.620369    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"]
Sep 16 12:20:16.621810 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:20:16.621786    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d62ba3d_5219_4153_b13a_ec54a22e919a.slice/crio-1f06db215169da985d30308388ac78e17a9da5fedd124dc72971df35222e12fe WatchSource:0}: Error finding container 1f06db215169da985d30308388ac78e17a9da5fedd124dc72971df35222e12fe: Status 404 returned error can't find the container with id 1f06db215169da985d30308388ac78e17a9da5fedd124dc72971df35222e12fe
Sep 16 12:20:17.356347 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:17.356305    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7" event={"ID":"3d62ba3d-5219-4153-b13a-ec54a22e919a","Type":"ContainerStarted","Data":"83b3beea9d1e836feec94f5559d3f9a7ca7b0f0ff46216cdfdc02b0567720a41"}
Sep 16 12:20:17.356518 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:17.356369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7" event={"ID":"3d62ba3d-5219-4153-b13a-ec54a22e919a","Type":"ContainerStarted","Data":"1f06db215169da985d30308388ac78e17a9da5fedd124dc72971df35222e12fe"}
Sep 16 12:20:17.356565 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:17.356523    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:20:17.380936 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:17.380835    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7" podStartSLOduration=1.891697988 podStartE2EDuration="2.38081763s" totalImagesPullingTime="489.119642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:17.379092027 +0000 UTC m=+1176.629629578" watchObservedRunningTime="2026-09-16 12:20:17.38081763 +0000 UTC m=+1176.631355178"
Sep 16 12:20:23.368252 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:23.368223    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:20:28.774518 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:28.774485    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"]
Sep 16 12:20:28.774969 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:28.774679    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk" podUID="7c61d966-b6b0-4602-8a20-289cc62dd512" containerName="maas-api" containerID="cri-o://4cb8ebf1416722dc2a62f2bb29ea87d02df8adc62ab5c888ac71eb26aa14cf08" gracePeriod=30
Sep 16 12:20:29.028272 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.028202    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:20:29.065368 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.065337    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkxlk\" (UniqueName: \"kubernetes.io/projected/7c61d966-b6b0-4602-8a20-289cc62dd512-kube-api-access-rkxlk\") pod \"7c61d966-b6b0-4602-8a20-289cc62dd512\" (UID: \"7c61d966-b6b0-4602-8a20-289cc62dd512\") "
Sep 16 12:20:29.065368 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.065369    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c61d966-b6b0-4602-8a20-289cc62dd512-maas-api-tls\") pod \"7c61d966-b6b0-4602-8a20-289cc62dd512\" (UID: \"7c61d966-b6b0-4602-8a20-289cc62dd512\") "
Sep 16 12:20:29.065620 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.065387    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c61d966-b6b0-4602-8a20-289cc62dd512-metrics-certs\") pod \"7c61d966-b6b0-4602-8a20-289cc62dd512\" (UID: \"7c61d966-b6b0-4602-8a20-289cc62dd512\") "
Sep 16 12:20:29.065620 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.065487    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c61d966-b6b0-4602-8a20-289cc62dd512-openshift-service-ca\") pod \"7c61d966-b6b0-4602-8a20-289cc62dd512\" (UID: \"7c61d966-b6b0-4602-8a20-289cc62dd512\") "
Sep 16 12:20:29.066068 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.066016    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c61d966-b6b0-4602-8a20-289cc62dd512-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c61d966-b6b0-4602-8a20-289cc62dd512" (UID: "7c61d966-b6b0-4602-8a20-289cc62dd512"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:20:29.067890 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.067859    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c61d966-b6b0-4602-8a20-289cc62dd512-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7c61d966-b6b0-4602-8a20-289cc62dd512" (UID: "7c61d966-b6b0-4602-8a20-289cc62dd512"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:20:29.067987 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.067912    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c61d966-b6b0-4602-8a20-289cc62dd512-kube-api-access-rkxlk" (OuterVolumeSpecName: "kube-api-access-rkxlk") pod "7c61d966-b6b0-4602-8a20-289cc62dd512" (UID: "7c61d966-b6b0-4602-8a20-289cc62dd512"). InnerVolumeSpecName "kube-api-access-rkxlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:20:29.068034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.068012    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c61d966-b6b0-4602-8a20-289cc62dd512-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c61d966-b6b0-4602-8a20-289cc62dd512" (UID: "7c61d966-b6b0-4602-8a20-289cc62dd512"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:20:29.166350 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.166317    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkxlk\" (UniqueName: \"kubernetes.io/projected/7c61d966-b6b0-4602-8a20-289cc62dd512-kube-api-access-rkxlk\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:20:29.166350 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.166350    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c61d966-b6b0-4602-8a20-289cc62dd512-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:20:29.166350 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.166358    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c61d966-b6b0-4602-8a20-289cc62dd512-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:20:29.166350 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.166366    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c61d966-b6b0-4602-8a20-289cc62dd512-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:20:29.431380 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.431342    2582 generic.go:356] "Generic (PLEG): container finished" podID="7c61d966-b6b0-4602-8a20-289cc62dd512" containerID="4cb8ebf1416722dc2a62f2bb29ea87d02df8adc62ab5c888ac71eb26aa14cf08" exitCode=0
Sep 16 12:20:29.431589 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.431392    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"
Sep 16 12:20:29.431589 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.431432    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk" event={"ID":"7c61d966-b6b0-4602-8a20-289cc62dd512","Type":"ContainerDied","Data":"4cb8ebf1416722dc2a62f2bb29ea87d02df8adc62ab5c888ac71eb26aa14cf08"}
Sep 16 12:20:29.431589 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.431487    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk" event={"ID":"7c61d966-b6b0-4602-8a20-289cc62dd512","Type":"ContainerDied","Data":"80a40312481562739d010236528917a592689012e8756ba7fb91e7ccd8566a54"}
Sep 16 12:20:29.431589 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.431511    2582 scope.go:117] "RemoveContainer" containerID="4cb8ebf1416722dc2a62f2bb29ea87d02df8adc62ab5c888ac71eb26aa14cf08"
Sep 16 12:20:29.440651 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.440630    2582 scope.go:117] "RemoveContainer" containerID="4cb8ebf1416722dc2a62f2bb29ea87d02df8adc62ab5c888ac71eb26aa14cf08"
Sep 16 12:20:29.440964 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:20:29.440945    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cb8ebf1416722dc2a62f2bb29ea87d02df8adc62ab5c888ac71eb26aa14cf08\": container with ID starting with 4cb8ebf1416722dc2a62f2bb29ea87d02df8adc62ab5c888ac71eb26aa14cf08 not found: ID does not exist" containerID="4cb8ebf1416722dc2a62f2bb29ea87d02df8adc62ab5c888ac71eb26aa14cf08"
Sep 16 12:20:29.441035 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.440972    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cb8ebf1416722dc2a62f2bb29ea87d02df8adc62ab5c888ac71eb26aa14cf08"} err="failed to get container status \"4cb8ebf1416722dc2a62f2bb29ea87d02df8adc62ab5c888ac71eb26aa14cf08\": rpc error: code = NotFound desc = could not find container \"4cb8ebf1416722dc2a62f2bb29ea87d02df8adc62ab5c888ac71eb26aa14cf08\": container with ID starting with 4cb8ebf1416722dc2a62f2bb29ea87d02df8adc62ab5c888ac71eb26aa14cf08 not found: ID does not exist"
Sep 16 12:20:29.447792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.447760    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"]
Sep 16 12:20:29.454476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:29.454454    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ce21877-6d57f5cf4f-z6wkk"]
Sep 16 12:20:31.212607 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:31.212573    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c61d966-b6b0-4602-8a20-289cc62dd512" path="/var/lib/kubelet/pods/7c61d966-b6b0-4602-8a20-289cc62dd512/volumes"
Sep 16 12:20:35.000579 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.000547    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"]
Sep 16 12:20:35.001148 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.000967    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c61d966-b6b0-4602-8a20-289cc62dd512" containerName="maas-api"
Sep 16 12:20:35.001148 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.000980    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c61d966-b6b0-4602-8a20-289cc62dd512" containerName="maas-api"
Sep 16 12:20:35.001148 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.001049    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c61d966-b6b0-4602-8a20-289cc62dd512" containerName="maas-api"
Sep 16 12:20:35.005306 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.005281    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:20:35.008028 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.008003    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b6f4a170-serving-cert\""
Sep 16 12:20:35.016650 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.016625    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"]
Sep 16 12:20:35.121481 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.121441    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-maas-api-tls\") pod \"maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw\" (UID: \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:20:35.121667 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.121494    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-openshift-service-ca\") pod \"maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw\" (UID: \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:20:35.121667 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.121594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klvkd\" (UniqueName: \"kubernetes.io/projected/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-kube-api-access-klvkd\") pod \"maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw\" (UID: \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:20:35.121851 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.121676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-metrics-certs\") pod \"maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw\" (UID: \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:20:35.222158 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.222127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-maas-api-tls\") pod \"maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw\" (UID: \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:20:35.222335 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.222175    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-openshift-service-ca\") pod \"maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw\" (UID: \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:20:35.222335 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.222248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klvkd\" (UniqueName: \"kubernetes.io/projected/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-kube-api-access-klvkd\") pod \"maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw\" (UID: \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:20:35.222335 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:20:35.222278    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-serving-cert: secret "maas-api-e2e-migrate-b6f4a170-serving-cert" not found
Sep 16 12:20:35.222335 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.222322    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-metrics-certs\") pod \"maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw\" (UID: \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:20:35.222596 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:20:35.222345    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-maas-api-tls podName:bc267c6e-aebd-4ae4-8da5-9f76fa7b110a nodeName:}" failed. No retries permitted until 2026-09-16 12:20:35.72232337 +0000 UTC m=+1194.972860897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-maas-api-tls") pod "maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw" (UID: "bc267c6e-aebd-4ae4-8da5-9f76fa7b110a") : secret "maas-api-e2e-migrate-b6f4a170-serving-cert" not found
Sep 16 12:20:35.222950 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.222931    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-openshift-service-ca\") pod \"maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw\" (UID: \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:20:35.225050 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.225026    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-metrics-certs\") pod \"maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw\" (UID: \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:20:35.231604 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.231576    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klvkd\" (UniqueName: \"kubernetes.io/projected/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-kube-api-access-klvkd\") pod \"maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw\" (UID: \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:20:35.728284 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.728240    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-maas-api-tls\") pod \"maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw\" (UID: \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:20:35.730963 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.730937    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-maas-api-tls\") pod \"maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw\" (UID: \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:20:35.918570 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:35.918527    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:20:36.194810 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:36.194781    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"]
Sep 16 12:20:36.195849 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:20:36.195817    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc267c6e_aebd_4ae4_8da5_9f76fa7b110a.slice/crio-f8f960835b121ff0340811e03a7d68b537a68b51f2439ec1b91593e4bcf34c8d WatchSource:0}: Error finding container f8f960835b121ff0340811e03a7d68b537a68b51f2439ec1b91593e4bcf34c8d: Status 404 returned error can't find the container with id f8f960835b121ff0340811e03a7d68b537a68b51f2439ec1b91593e4bcf34c8d
Sep 16 12:20:36.470450 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:36.470368    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw" event={"ID":"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a","Type":"ContainerStarted","Data":"f8f960835b121ff0340811e03a7d68b537a68b51f2439ec1b91593e4bcf34c8d"}
Sep 16 12:20:37.478324 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:37.478289    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw" event={"ID":"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a","Type":"ContainerStarted","Data":"022f51243a21ab69c2af3635fa26ba207e4ea4655e03215ce15c4b751574a444"}
Sep 16 12:20:37.478812 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:37.478429    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:20:37.495964 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:37.495912    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw" podStartSLOduration=3.193579066 podStartE2EDuration="3.495896419s" totalImagesPullingTime="302.317353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:37.49479087 +0000 UTC m=+1196.745328424" watchObservedRunningTime="2026-09-16 12:20:37.495896419 +0000 UTC m=+1196.746433964"
Sep 16 12:20:43.489817 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:43.489780    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:20:53.094408 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.094375    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"]
Sep 16 12:20:53.097874 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.097857    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.102595 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.102570    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7a4d5c\"/\"kube-root-ca.crt\""
Sep 16 12:20:53.102777 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.102570    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7a4d5c\"/\"openshift-service-ca.crt\""
Sep 16 12:20:53.102777 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.102571    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7a4d5c\"/\"default-dockercfg-mml7v\""
Sep 16 12:20:53.102777 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.102571    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7a4d5c\"/\"simulator-w4-7a4d5c-kserve-self-signed-certs\""
Sep 16 12:20:53.106709 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.106683    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"]
Sep 16 12:20:53.199403 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.199364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d662x\" (UniqueName: \"kubernetes.io/projected/e20e88fc-24e6-4ba3-b75b-e90da065f515-kube-api-access-d662x\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.199593 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.199419    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-home\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.199593 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.199442    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e20e88fc-24e6-4ba3-b75b-e90da065f515-tls-certs\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.199593 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.199519    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-dshm\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.199593 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.199579    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-model-cache\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.199793 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.199614    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-tmp-dir\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.300826 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.300780    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d662x\" (UniqueName: \"kubernetes.io/projected/e20e88fc-24e6-4ba3-b75b-e90da065f515-kube-api-access-d662x\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.301032 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.300856    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-home\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.301032 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.300890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e20e88fc-24e6-4ba3-b75b-e90da065f515-tls-certs\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.301032 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.300915    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-dshm\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.301032 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.300957    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-model-cache\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.301032 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.300999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-tmp-dir\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.301327 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.301304    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-home\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.301395 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.301357    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-model-cache\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.301522 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.301503    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-tmp-dir\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.303539 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.303521    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-dshm\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.303724 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.303706    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e20e88fc-24e6-4ba3-b75b-e90da065f515-tls-certs\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.309902 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.309874    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d662x\" (UniqueName: \"kubernetes.io/projected/e20e88fc-24e6-4ba3-b75b-e90da065f515-kube-api-access-d662x\") pod \"simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") " pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.408585 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.408498    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:20:53.690179 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:53.690144    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"]
Sep 16 12:20:53.691698 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:20:53.691654    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode20e88fc_24e6_4ba3_b75b_e90da065f515.slice/crio-ae249147558fed85dfe02b22bc1730b175f6ba544537c2c21706ba6347cf2510 WatchSource:0}: Error finding container ae249147558fed85dfe02b22bc1730b175f6ba544537c2c21706ba6347cf2510: Status 404 returned error can't find the container with id ae249147558fed85dfe02b22bc1730b175f6ba544537c2c21706ba6347cf2510
Sep 16 12:20:54.575683 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:54.575641    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw" event={"ID":"e20e88fc-24e6-4ba3-b75b-e90da065f515","Type":"ContainerStarted","Data":"ee01c8358dfad792afc40a955d6212097649003ec5bdc7c574a83d123d7865fa"}
Sep 16 12:20:54.576175 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:54.575691    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw" event={"ID":"e20e88fc-24e6-4ba3-b75b-e90da065f515","Type":"ContainerStarted","Data":"ae249147558fed85dfe02b22bc1730b175f6ba544537c2c21706ba6347cf2510"}
Sep 16 12:20:54.595072 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:20:54.595012    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw" podStartSLOduration=1.594992826 podStartE2EDuration="1.594992826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:54.593424579 +0000 UTC m=+1213.843962128" watchObservedRunningTime="2026-09-16 12:20:54.594992826 +0000 UTC m=+1213.845530379"
Sep 16 12:21:03.409078 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:03.409034    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:21:03.409078 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:03.409085    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:21:03.422095 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:03.422069    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:21:03.638287 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:03.638260    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:21:04.134208 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.134178    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"]
Sep 16 12:21:04.134407 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.134385    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw" podUID="bc267c6e-aebd-4ae4-8da5-9f76fa7b110a" containerName="maas-api" containerID="cri-o://022f51243a21ab69c2af3635fa26ba207e4ea4655e03215ce15c4b751574a444" gracePeriod=30
Sep 16 12:21:04.386433 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.386375    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:21:04.510252 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.510218    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-openshift-service-ca\") pod \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\" (UID: \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\") "
Sep 16 12:21:04.510756 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.510279    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-maas-api-tls\") pod \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\" (UID: \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\") "
Sep 16 12:21:04.510756 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.510367    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-metrics-certs\") pod \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\" (UID: \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\") "
Sep 16 12:21:04.510756 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.510410    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klvkd\" (UniqueName: \"kubernetes.io/projected/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-kube-api-access-klvkd\") pod \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\" (UID: \"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a\") "
Sep 16 12:21:04.510756 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.510648    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bc267c6e-aebd-4ae4-8da5-9f76fa7b110a" (UID: "bc267c6e-aebd-4ae4-8da5-9f76fa7b110a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:21:04.510979 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.510803    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:21:04.524524 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.523705    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bc267c6e-aebd-4ae4-8da5-9f76fa7b110a" (UID: "bc267c6e-aebd-4ae4-8da5-9f76fa7b110a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:21:04.524524 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.524096    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bc267c6e-aebd-4ae4-8da5-9f76fa7b110a" (UID: "bc267c6e-aebd-4ae4-8da5-9f76fa7b110a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:21:04.525425 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.525392    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-kube-api-access-klvkd" (OuterVolumeSpecName: "kube-api-access-klvkd") pod "bc267c6e-aebd-4ae4-8da5-9f76fa7b110a" (UID: "bc267c6e-aebd-4ae4-8da5-9f76fa7b110a"). InnerVolumeSpecName "kube-api-access-klvkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:21:04.612249 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.612188    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:21:04.612249 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.612243    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klvkd\" (UniqueName: \"kubernetes.io/projected/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-kube-api-access-klvkd\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:21:04.612249 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.612255    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:21:04.633802 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.633771    2582 generic.go:356] "Generic (PLEG): container finished" podID="bc267c6e-aebd-4ae4-8da5-9f76fa7b110a" containerID="022f51243a21ab69c2af3635fa26ba207e4ea4655e03215ce15c4b751574a444" exitCode=0
Sep 16 12:21:04.633947 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.633826    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"
Sep 16 12:21:04.633947 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.633861    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw" event={"ID":"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a","Type":"ContainerDied","Data":"022f51243a21ab69c2af3635fa26ba207e4ea4655e03215ce15c4b751574a444"}
Sep 16 12:21:04.634022 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.633944    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw" event={"ID":"bc267c6e-aebd-4ae4-8da5-9f76fa7b110a","Type":"ContainerDied","Data":"f8f960835b121ff0340811e03a7d68b537a68b51f2439ec1b91593e4bcf34c8d"}
Sep 16 12:21:04.634022 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.633975    2582 scope.go:117] "RemoveContainer" containerID="022f51243a21ab69c2af3635fa26ba207e4ea4655e03215ce15c4b751574a444"
Sep 16 12:21:04.644054 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.644034    2582 scope.go:117] "RemoveContainer" containerID="022f51243a21ab69c2af3635fa26ba207e4ea4655e03215ce15c4b751574a444"
Sep 16 12:21:04.644320 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:21:04.644302    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"022f51243a21ab69c2af3635fa26ba207e4ea4655e03215ce15c4b751574a444\": container with ID starting with 022f51243a21ab69c2af3635fa26ba207e4ea4655e03215ce15c4b751574a444 not found: ID does not exist" containerID="022f51243a21ab69c2af3635fa26ba207e4ea4655e03215ce15c4b751574a444"
Sep 16 12:21:04.644374 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.644329    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"022f51243a21ab69c2af3635fa26ba207e4ea4655e03215ce15c4b751574a444"} err="failed to get container status \"022f51243a21ab69c2af3635fa26ba207e4ea4655e03215ce15c4b751574a444\": rpc error: code = NotFound desc = could not find container \"022f51243a21ab69c2af3635fa26ba207e4ea4655e03215ce15c4b751574a444\": container with ID starting with 022f51243a21ab69c2af3635fa26ba207e4ea4655e03215ce15c4b751574a444 not found: ID does not exist"
Sep 16 12:21:04.656264 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.656244    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"]
Sep 16 12:21:04.659414 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:04.659387    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6f4a170-74bdffb587-qv7zw"]
Sep 16 12:21:05.214814 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:05.214701    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc267c6e-aebd-4ae4-8da5-9f76fa7b110a" path="/var/lib/kubelet/pods/bc267c6e-aebd-4ae4-8da5-9f76fa7b110a/volumes"
Sep 16 12:21:09.597346 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.597312    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"]
Sep 16 12:21:09.597787 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.597684    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc267c6e-aebd-4ae4-8da5-9f76fa7b110a" containerName="maas-api"
Sep 16 12:21:09.597787 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.597696    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc267c6e-aebd-4ae4-8da5-9f76fa7b110a" containerName="maas-api"
Sep 16 12:21:09.597871 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.597812    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc267c6e-aebd-4ae4-8da5-9f76fa7b110a" containerName="maas-api"
Sep 16 12:21:09.601341 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.601306    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:09.604055 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.604032    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f6101af5-serving-cert\""
Sep 16 12:21:09.614347 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.614317    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"]
Sep 16 12:21:09.660031 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.659996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-maas-api-tls\") pod \"maas-api-e2e-ait-f6101af5-866997b58c-bgs89\" (UID: \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:09.660212 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.660129    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lrwz\" (UniqueName: \"kubernetes.io/projected/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-kube-api-access-4lrwz\") pod \"maas-api-e2e-ait-f6101af5-866997b58c-bgs89\" (UID: \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:09.660268 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.660209    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-openshift-service-ca\") pod \"maas-api-e2e-ait-f6101af5-866997b58c-bgs89\" (UID: \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:09.660268 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.660244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-metrics-certs\") pod \"maas-api-e2e-ait-f6101af5-866997b58c-bgs89\" (UID: \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:09.761669 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.761623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-openshift-service-ca\") pod \"maas-api-e2e-ait-f6101af5-866997b58c-bgs89\" (UID: \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:09.761669 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.761667    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-metrics-certs\") pod \"maas-api-e2e-ait-f6101af5-866997b58c-bgs89\" (UID: \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:09.761908 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.761688    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-maas-api-tls\") pod \"maas-api-e2e-ait-f6101af5-866997b58c-bgs89\" (UID: \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:09.761908 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.761758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrwz\" (UniqueName: \"kubernetes.io/projected/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-kube-api-access-4lrwz\") pod \"maas-api-e2e-ait-f6101af5-866997b58c-bgs89\" (UID: \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:09.762418 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.762391    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-openshift-service-ca\") pod \"maas-api-e2e-ait-f6101af5-866997b58c-bgs89\" (UID: \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:09.764552 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.764525    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-maas-api-tls\") pod \"maas-api-e2e-ait-f6101af5-866997b58c-bgs89\" (UID: \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:09.764651 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.764573    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-metrics-certs\") pod \"maas-api-e2e-ait-f6101af5-866997b58c-bgs89\" (UID: \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:09.770069 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.770050    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lrwz\" (UniqueName: \"kubernetes.io/projected/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-kube-api-access-4lrwz\") pod \"maas-api-e2e-ait-f6101af5-866997b58c-bgs89\" (UID: \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:09.914012 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:09.913920    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:10.189112 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:10.189089    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"]
Sep 16 12:21:10.190833 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:21:10.190799    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2f25fc_752b_47cf_ae9b_2c04e059ffac.slice/crio-fee7c0de9995d43c8b704d9864a78f4e0109739edf43716cd3f7e55d28541162 WatchSource:0}: Error finding container fee7c0de9995d43c8b704d9864a78f4e0109739edf43716cd3f7e55d28541162: Status 404 returned error can't find the container with id fee7c0de9995d43c8b704d9864a78f4e0109739edf43716cd3f7e55d28541162
Sep 16 12:21:10.679587 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:10.679544    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89" event={"ID":"7e2f25fc-752b-47cf-ae9b-2c04e059ffac","Type":"ContainerStarted","Data":"b8b636353c4883a5890808267d4d0fd46bae6ec9f1844d76394af99bf626d83a"}
Sep 16 12:21:10.680104 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:10.679646    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89" event={"ID":"7e2f25fc-752b-47cf-ae9b-2c04e059ffac","Type":"ContainerStarted","Data":"fee7c0de9995d43c8b704d9864a78f4e0109739edf43716cd3f7e55d28541162"}
Sep 16 12:21:10.680104 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:10.679675    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:10.738030 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:10.737974    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89" podStartSLOduration=1.323496574 podStartE2EDuration="1.7379572s" totalImagesPullingTime="414.460626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:21:10.736659405 +0000 UTC m=+1229.987196967" watchObservedRunningTime="2026-09-16 12:21:10.7379572 +0000 UTC m=+1229.988494802"
Sep 16 12:21:16.690859 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:16.690778    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:39.203968 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.203882    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"]
Sep 16 12:21:39.204485 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.204173    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89" podUID="7e2f25fc-752b-47cf-ae9b-2c04e059ffac" containerName="maas-api" containerID="cri-o://b8b636353c4883a5890808267d4d0fd46bae6ec9f1844d76394af99bf626d83a" gracePeriod=30
Sep 16 12:21:39.450640 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.450608    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:39.548863 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.548771    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-maas-api-tls\") pod \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\" (UID: \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\") "
Sep 16 12:21:39.548863 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.548828    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lrwz\" (UniqueName: \"kubernetes.io/projected/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-kube-api-access-4lrwz\") pod \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\" (UID: \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\") "
Sep 16 12:21:39.549109 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.548951    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-metrics-certs\") pod \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\" (UID: \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\") "
Sep 16 12:21:39.549109 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.549001    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-openshift-service-ca\") pod \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\" (UID: \"7e2f25fc-752b-47cf-ae9b-2c04e059ffac\") "
Sep 16 12:21:39.549443 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.549409    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7e2f25fc-752b-47cf-ae9b-2c04e059ffac" (UID: "7e2f25fc-752b-47cf-ae9b-2c04e059ffac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:21:39.551203 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.551174    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-kube-api-access-4lrwz" (OuterVolumeSpecName: "kube-api-access-4lrwz") pod "7e2f25fc-752b-47cf-ae9b-2c04e059ffac" (UID: "7e2f25fc-752b-47cf-ae9b-2c04e059ffac"). InnerVolumeSpecName "kube-api-access-4lrwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:21:39.551308 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.551200    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7e2f25fc-752b-47cf-ae9b-2c04e059ffac" (UID: "7e2f25fc-752b-47cf-ae9b-2c04e059ffac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:21:39.551308 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.551226    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7e2f25fc-752b-47cf-ae9b-2c04e059ffac" (UID: "7e2f25fc-752b-47cf-ae9b-2c04e059ffac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:21:39.650276 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.650230    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:21:39.650276 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.650271    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lrwz\" (UniqueName: \"kubernetes.io/projected/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-kube-api-access-4lrwz\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:21:39.650276 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.650281    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:21:39.650276 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.650289    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2f25fc-752b-47cf-ae9b-2c04e059ffac-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:21:39.842448 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.842412    2582 generic.go:356] "Generic (PLEG): container finished" podID="7e2f25fc-752b-47cf-ae9b-2c04e059ffac" containerID="b8b636353c4883a5890808267d4d0fd46bae6ec9f1844d76394af99bf626d83a" exitCode=0
Sep 16 12:21:39.842634 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.842453    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"
Sep 16 12:21:39.842634 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.842497    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89" event={"ID":"7e2f25fc-752b-47cf-ae9b-2c04e059ffac","Type":"ContainerDied","Data":"b8b636353c4883a5890808267d4d0fd46bae6ec9f1844d76394af99bf626d83a"}
Sep 16 12:21:39.842634 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.842546    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89" event={"ID":"7e2f25fc-752b-47cf-ae9b-2c04e059ffac","Type":"ContainerDied","Data":"fee7c0de9995d43c8b704d9864a78f4e0109739edf43716cd3f7e55d28541162"}
Sep 16 12:21:39.842634 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.842571    2582 scope.go:117] "RemoveContainer" containerID="b8b636353c4883a5890808267d4d0fd46bae6ec9f1844d76394af99bf626d83a"
Sep 16 12:21:39.852629 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.852608    2582 scope.go:117] "RemoveContainer" containerID="b8b636353c4883a5890808267d4d0fd46bae6ec9f1844d76394af99bf626d83a"
Sep 16 12:21:39.852947 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:21:39.852928    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b636353c4883a5890808267d4d0fd46bae6ec9f1844d76394af99bf626d83a\": container with ID starting with b8b636353c4883a5890808267d4d0fd46bae6ec9f1844d76394af99bf626d83a not found: ID does not exist" containerID="b8b636353c4883a5890808267d4d0fd46bae6ec9f1844d76394af99bf626d83a"
Sep 16 12:21:39.853028 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.852956    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b636353c4883a5890808267d4d0fd46bae6ec9f1844d76394af99bf626d83a"} err="failed to get container status \"b8b636353c4883a5890808267d4d0fd46bae6ec9f1844d76394af99bf626d83a\": rpc error: code = NotFound desc = could not find container \"b8b636353c4883a5890808267d4d0fd46bae6ec9f1844d76394af99bf626d83a\": container with ID starting with b8b636353c4883a5890808267d4d0fd46bae6ec9f1844d76394af99bf626d83a not found: ID does not exist"
Sep 16 12:21:39.865232 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.865208    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"]
Sep 16 12:21:39.868678 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:39.868657    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f6101af5-866997b58c-bgs89"]
Sep 16 12:21:41.213421 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:41.213384    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e2f25fc-752b-47cf-ae9b-2c04e059ffac" path="/var/lib/kubelet/pods/7e2f25fc-752b-47cf-ae9b-2c04e059ffac/volumes"
Sep 16 12:21:48.389390 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.389348    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"]
Sep 16 12:21:48.389839 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.389811    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e2f25fc-752b-47cf-ae9b-2c04e059ffac" containerName="maas-api"
Sep 16 12:21:48.389839 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.389825    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e2f25fc-752b-47cf-ae9b-2c04e059ffac" containerName="maas-api"
Sep 16 12:21:48.389942 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.389903    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e2f25fc-752b-47cf-ae9b-2c04e059ffac" containerName="maas-api"
Sep 16 12:21:48.394762 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.394722    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.399315 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.399289    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0e8ffe\"/\"kube-root-ca.crt\""
Sep 16 12:21:48.399315 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.399309    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0e8ffe\"/\"simulator-w0-0e8ffe-kserve-self-signed-certs\""
Sep 16 12:21:48.399547 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.399358    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0e8ffe\"/\"default-dockercfg-5mkbg\""
Sep 16 12:21:48.399547 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.399378    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0e8ffe\"/\"openshift-service-ca.crt\""
Sep 16 12:21:48.405391 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.405364    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"]
Sep 16 12:21:48.536633 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.536591    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c685a655-d0c4-4f1d-87d0-130974c523fd-tls-certs\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.536633 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.536638    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-model-cache\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.536952 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.536657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-tmp-dir\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.536952 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.536805    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvd9m\" (UniqueName: \"kubernetes.io/projected/c685a655-d0c4-4f1d-87d0-130974c523fd-kube-api-access-jvd9m\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.536952 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.536897    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-dshm\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.536952 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.536939    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-home\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.637898 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.637859    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-dshm\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.637898 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.637905    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-home\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.638170 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.637942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c685a655-d0c4-4f1d-87d0-130974c523fd-tls-certs\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.638170 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.637967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-model-cache\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.638170 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.637986    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-tmp-dir\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.638170 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.638018    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvd9m\" (UniqueName: \"kubernetes.io/projected/c685a655-d0c4-4f1d-87d0-130974c523fd-kube-api-access-jvd9m\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.638448 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.638419    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-tmp-dir\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.638448 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.638444    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-home\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.638615 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.638455    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-model-cache\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.640397 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.640343    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-dshm\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.640655 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.640636    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c685a655-d0c4-4f1d-87d0-130974c523fd-tls-certs\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.646262 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.646242    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvd9m\" (UniqueName: \"kubernetes.io/projected/c685a655-d0c4-4f1d-87d0-130974c523fd-kube-api-access-jvd9m\") pod \"simulator-w0-0e8ffe-kserve-695b5bb49-99gvx\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") " pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.705433 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.705387    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:48.791381 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.791347    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"]
Sep 16 12:21:48.795201 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.795176    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:21:48.798440 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.798418    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-d6af8fa6-serving-cert\""
Sep 16 12:21:48.807919 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.807885    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"]
Sep 16 12:21:48.940870 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.940770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3107a49-9748-4a44-820b-7ab2130aee6d-metrics-certs\") pod \"maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw\" (UID: \"f3107a49-9748-4a44-820b-7ab2130aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:21:48.941051 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.940908    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3107a49-9748-4a44-820b-7ab2130aee6d-openshift-service-ca\") pod \"maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw\" (UID: \"f3107a49-9748-4a44-820b-7ab2130aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:21:48.941051 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.940952    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3107a49-9748-4a44-820b-7ab2130aee6d-maas-api-tls\") pod \"maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw\" (UID: \"f3107a49-9748-4a44-820b-7ab2130aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:21:48.941051 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.940991    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvkjh\" (UniqueName: \"kubernetes.io/projected/f3107a49-9748-4a44-820b-7ab2130aee6d-kube-api-access-vvkjh\") pod \"maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw\" (UID: \"f3107a49-9748-4a44-820b-7ab2130aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:21:48.979893 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:48.979857    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"]
Sep 16 12:21:48.982650 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:21:48.982620    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc685a655_d0c4_4f1d_87d0_130974c523fd.slice/crio-8f1a5010ccfd0bd21458d7fd846e08b2905fc8e1a5c0ef5ed6caf2354c0389b0 WatchSource:0}: Error finding container 8f1a5010ccfd0bd21458d7fd846e08b2905fc8e1a5c0ef5ed6caf2354c0389b0: Status 404 returned error can't find the container with id 8f1a5010ccfd0bd21458d7fd846e08b2905fc8e1a5c0ef5ed6caf2354c0389b0
Sep 16 12:21:49.041585 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:49.041559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3107a49-9748-4a44-820b-7ab2130aee6d-openshift-service-ca\") pod \"maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw\" (UID: \"f3107a49-9748-4a44-820b-7ab2130aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:21:49.041703 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:49.041613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3107a49-9748-4a44-820b-7ab2130aee6d-maas-api-tls\") pod \"maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw\" (UID: \"f3107a49-9748-4a44-820b-7ab2130aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:21:49.041703 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:49.041660    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvkjh\" (UniqueName: \"kubernetes.io/projected/f3107a49-9748-4a44-820b-7ab2130aee6d-kube-api-access-vvkjh\") pod \"maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw\" (UID: \"f3107a49-9748-4a44-820b-7ab2130aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:21:49.041845 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:49.041702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3107a49-9748-4a44-820b-7ab2130aee6d-metrics-certs\") pod \"maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw\" (UID: \"f3107a49-9748-4a44-820b-7ab2130aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:21:49.041845 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:21:49.041784    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-serving-cert: secret "maas-api-e2e-derive-d6af8fa6-serving-cert" not found
Sep 16 12:21:49.041961 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:21:49.041846    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3107a49-9748-4a44-820b-7ab2130aee6d-maas-api-tls podName:f3107a49-9748-4a44-820b-7ab2130aee6d nodeName:}" failed. No retries permitted until 2026-09-16 12:21:49.541825099 +0000 UTC m=+1268.792362633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3107a49-9748-4a44-820b-7ab2130aee6d-maas-api-tls") pod "maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw" (UID: "f3107a49-9748-4a44-820b-7ab2130aee6d") : secret "maas-api-e2e-derive-d6af8fa6-serving-cert" not found
Sep 16 12:21:49.042259 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:49.042239    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3107a49-9748-4a44-820b-7ab2130aee6d-openshift-service-ca\") pod \"maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw\" (UID: \"f3107a49-9748-4a44-820b-7ab2130aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:21:49.044232 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:49.044213    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3107a49-9748-4a44-820b-7ab2130aee6d-metrics-certs\") pod \"maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw\" (UID: \"f3107a49-9748-4a44-820b-7ab2130aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:21:49.050930 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:49.050904    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvkjh\" (UniqueName: \"kubernetes.io/projected/f3107a49-9748-4a44-820b-7ab2130aee6d-kube-api-access-vvkjh\") pod \"maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw\" (UID: \"f3107a49-9748-4a44-820b-7ab2130aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:21:49.546903 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:49.546872    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3107a49-9748-4a44-820b-7ab2130aee6d-maas-api-tls\") pod \"maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw\" (UID: \"f3107a49-9748-4a44-820b-7ab2130aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:21:49.549516 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:49.549495    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3107a49-9748-4a44-820b-7ab2130aee6d-maas-api-tls\") pod \"maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw\" (UID: \"f3107a49-9748-4a44-820b-7ab2130aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:21:49.708833 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:49.708796    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:21:49.903408 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:49.903373    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx" event={"ID":"c685a655-d0c4-4f1d-87d0-130974c523fd","Type":"ContainerStarted","Data":"07161ce81535fd8432e6f3e8cc3b1544211886bb36577ebf5f1deda6de625147"}
Sep 16 12:21:49.903408 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:49.903412    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx" event={"ID":"c685a655-d0c4-4f1d-87d0-130974c523fd","Type":"ContainerStarted","Data":"8f1a5010ccfd0bd21458d7fd846e08b2905fc8e1a5c0ef5ed6caf2354c0389b0"}
Sep 16 12:21:49.922865 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:49.922809    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx" podStartSLOduration=1.922793468 podStartE2EDuration="1.922793468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:21:49.920312257 +0000 UTC m=+1269.170849807" watchObservedRunningTime="2026-09-16 12:21:49.922793468 +0000 UTC m=+1269.173331067"
Sep 16 12:21:49.981686 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:49.981656    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"]
Sep 16 12:21:49.983011 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:21:49.982985    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3107a49_9748_4a44_820b_7ab2130aee6d.slice/crio-2222d92964be668c35940a01b67b55634a547509f82b0e85d5b50f001cf5794a WatchSource:0}: Error finding container 2222d92964be668c35940a01b67b55634a547509f82b0e85d5b50f001cf5794a: Status 404 returned error can't find the container with id 2222d92964be668c35940a01b67b55634a547509f82b0e85d5b50f001cf5794a
Sep 16 12:21:50.911847 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:50.911806    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw" event={"ID":"f3107a49-9748-4a44-820b-7ab2130aee6d","Type":"ContainerStarted","Data":"8ac7f6d90952257be0d1d2a9df13b393b7d191543accb92bd6853e7b7d8882ec"}
Sep 16 12:21:50.912269 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:50.911878    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw" event={"ID":"f3107a49-9748-4a44-820b-7ab2130aee6d","Type":"ContainerStarted","Data":"2222d92964be668c35940a01b67b55634a547509f82b0e85d5b50f001cf5794a"}
Sep 16 12:21:50.912269 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:50.911972    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:21:50.931658 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:50.931601    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw" podStartSLOduration=2.570464103 podStartE2EDuration="2.93158483s" totalImagesPullingTime="361.120727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:21:50.929784348 +0000 UTC m=+1270.180321898" watchObservedRunningTime="2026-09-16 12:21:50.93158483 +0000 UTC m=+1270.182122414"
Sep 16 12:21:56.924061 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:56.924033    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:21:58.706111 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:58.706067    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:58.706613 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:58.706128    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:58.719939 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:58.719907    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:21:58.972107 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:21:58.972018    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:22:24.211302 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.211266    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"]
Sep 16 12:22:24.215345 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.215322    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.218497 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.218472    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7c3429\"/\"openshift-service-ca.crt\""
Sep 16 12:22:24.220010 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.219989    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7c3429\"/\"autores-model-7c3429-kserve-self-signed-certs\""
Sep 16 12:22:24.220185 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.220161    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7c3429\"/\"default-dockercfg-6pnmv\""
Sep 16 12:22:24.220410 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.220361    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7c3429\"/\"kube-root-ca.crt\""
Sep 16 12:22:24.228380 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.228354    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"]
Sep 16 12:22:24.267174 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.267131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5692b6cf-69ba-4249-9ec2-a38a7ba45372-tls-certs\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.267349 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.267197    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-home\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.267349 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.267228    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-tmp-dir\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.267349 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.267259    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfwv5\" (UniqueName: \"kubernetes.io/projected/5692b6cf-69ba-4249-9ec2-a38a7ba45372-kube-api-access-hfwv5\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.267349 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.267282    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-dshm\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.267528 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.267349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-model-cache\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.368684 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.368646    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-dshm\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.368684 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.368690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-model-cache\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.368939 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.368759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5692b6cf-69ba-4249-9ec2-a38a7ba45372-tls-certs\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.368939 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.368806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-home\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.368939 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.368849    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-tmp-dir\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.368939 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.368897    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwv5\" (UniqueName: \"kubernetes.io/projected/5692b6cf-69ba-4249-9ec2-a38a7ba45372-kube-api-access-hfwv5\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.369260 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.369224    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-model-cache\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.369378 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.369274    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-home\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.369378 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.369310    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-tmp-dir\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.371199 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.371177    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-dshm\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.371340 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.371324    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5692b6cf-69ba-4249-9ec2-a38a7ba45372-tls-certs\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.376987 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.376958    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfwv5\" (UniqueName: \"kubernetes.io/projected/5692b6cf-69ba-4249-9ec2-a38a7ba45372-kube-api-access-hfwv5\") pod \"autores-model-7c3429-kserve-5c6b89499f-hxvhk\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") " pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.527579 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.527480    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:24.799878 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:24.799845    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"]
Sep 16 12:22:24.801063 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:22:24.801031    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5692b6cf_69ba_4249_9ec2_a38a7ba45372.slice/crio-28975a622cc65db28cc67e5f1d51754ae692b5e110aca75d678d000ca33c56f6 WatchSource:0}: Error finding container 28975a622cc65db28cc67e5f1d51754ae692b5e110aca75d678d000ca33c56f6: Status 404 returned error can't find the container with id 28975a622cc65db28cc67e5f1d51754ae692b5e110aca75d678d000ca33c56f6
Sep 16 12:22:25.105013 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:25.104954    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk" event={"ID":"5692b6cf-69ba-4249-9ec2-a38a7ba45372","Type":"ContainerStarted","Data":"af324ed7b0b92cdaf35da26638d5300b4274d127cd50bc71d98ed9cb2c1341eb"}
Sep 16 12:22:25.105210 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:25.105035    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk" event={"ID":"5692b6cf-69ba-4249-9ec2-a38a7ba45372","Type":"ContainerStarted","Data":"28975a622cc65db28cc67e5f1d51754ae692b5e110aca75d678d000ca33c56f6"}
Sep 16 12:22:25.123151 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:25.123103    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk" podStartSLOduration=1.123086913 podStartE2EDuration="1.123086913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:22:25.120443768 +0000 UTC m=+1304.370981319" watchObservedRunningTime="2026-09-16 12:22:25.123086913 +0000 UTC m=+1304.373624475"
Sep 16 12:22:26.680343 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:26.680308    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"]
Sep 16 12:22:26.680869 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:26.680529    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw" podUID="f3107a49-9748-4a44-820b-7ab2130aee6d" containerName="maas-api" containerID="cri-o://8ac7f6d90952257be0d1d2a9df13b393b7d191543accb92bd6853e7b7d8882ec" gracePeriod=30
Sep 16 12:22:26.934930 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:26.934859    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:22:26.995385 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:26.995353    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3107a49-9748-4a44-820b-7ab2130aee6d-openshift-service-ca\") pod \"f3107a49-9748-4a44-820b-7ab2130aee6d\" (UID: \"f3107a49-9748-4a44-820b-7ab2130aee6d\") "
Sep 16 12:22:26.995588 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:26.995410    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3107a49-9748-4a44-820b-7ab2130aee6d-metrics-certs\") pod \"f3107a49-9748-4a44-820b-7ab2130aee6d\" (UID: \"f3107a49-9748-4a44-820b-7ab2130aee6d\") "
Sep 16 12:22:26.995588 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:26.995475    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvkjh\" (UniqueName: \"kubernetes.io/projected/f3107a49-9748-4a44-820b-7ab2130aee6d-kube-api-access-vvkjh\") pod \"f3107a49-9748-4a44-820b-7ab2130aee6d\" (UID: \"f3107a49-9748-4a44-820b-7ab2130aee6d\") "
Sep 16 12:22:26.995588 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:26.995539    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3107a49-9748-4a44-820b-7ab2130aee6d-maas-api-tls\") pod \"f3107a49-9748-4a44-820b-7ab2130aee6d\" (UID: \"f3107a49-9748-4a44-820b-7ab2130aee6d\") "
Sep 16 12:22:26.995803 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:26.995694    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3107a49-9748-4a44-820b-7ab2130aee6d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f3107a49-9748-4a44-820b-7ab2130aee6d" (UID: "f3107a49-9748-4a44-820b-7ab2130aee6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:22:26.997777 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:26.997722    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3107a49-9748-4a44-820b-7ab2130aee6d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f3107a49-9748-4a44-820b-7ab2130aee6d" (UID: "f3107a49-9748-4a44-820b-7ab2130aee6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:22:26.997898 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:26.997802    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3107a49-9748-4a44-820b-7ab2130aee6d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f3107a49-9748-4a44-820b-7ab2130aee6d" (UID: "f3107a49-9748-4a44-820b-7ab2130aee6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:22:26.997898 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:26.997819    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3107a49-9748-4a44-820b-7ab2130aee6d-kube-api-access-vvkjh" (OuterVolumeSpecName: "kube-api-access-vvkjh") pod "f3107a49-9748-4a44-820b-7ab2130aee6d" (UID: "f3107a49-9748-4a44-820b-7ab2130aee6d"). InnerVolumeSpecName "kube-api-access-vvkjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:22:27.096908 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:27.096875    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvkjh\" (UniqueName: \"kubernetes.io/projected/f3107a49-9748-4a44-820b-7ab2130aee6d-kube-api-access-vvkjh\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:22:27.096908 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:27.096905    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3107a49-9748-4a44-820b-7ab2130aee6d-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:22:27.096908 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:27.096914    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3107a49-9748-4a44-820b-7ab2130aee6d-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:22:27.097148 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:27.096924    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3107a49-9748-4a44-820b-7ab2130aee6d-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:22:27.118216 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:27.118178    2582 generic.go:356] "Generic (PLEG): container finished" podID="f3107a49-9748-4a44-820b-7ab2130aee6d" containerID="8ac7f6d90952257be0d1d2a9df13b393b7d191543accb92bd6853e7b7d8882ec" exitCode=0
Sep 16 12:22:27.118459 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:27.118226    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"
Sep 16 12:22:27.118459 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:27.118260    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw" event={"ID":"f3107a49-9748-4a44-820b-7ab2130aee6d","Type":"ContainerDied","Data":"8ac7f6d90952257be0d1d2a9df13b393b7d191543accb92bd6853e7b7d8882ec"}
Sep 16 12:22:27.118459 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:27.118336    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw" event={"ID":"f3107a49-9748-4a44-820b-7ab2130aee6d","Type":"ContainerDied","Data":"2222d92964be668c35940a01b67b55634a547509f82b0e85d5b50f001cf5794a"}
Sep 16 12:22:27.118459 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:27.118379    2582 scope.go:117] "RemoveContainer" containerID="8ac7f6d90952257be0d1d2a9df13b393b7d191543accb92bd6853e7b7d8882ec"
Sep 16 12:22:27.130984 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:27.130960    2582 scope.go:117] "RemoveContainer" containerID="8ac7f6d90952257be0d1d2a9df13b393b7d191543accb92bd6853e7b7d8882ec"
Sep 16 12:22:27.131481 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:22:27.131459    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ac7f6d90952257be0d1d2a9df13b393b7d191543accb92bd6853e7b7d8882ec\": container with ID starting with 8ac7f6d90952257be0d1d2a9df13b393b7d191543accb92bd6853e7b7d8882ec not found: ID does not exist" containerID="8ac7f6d90952257be0d1d2a9df13b393b7d191543accb92bd6853e7b7d8882ec"
Sep 16 12:22:27.131574 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:27.131493    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac7f6d90952257be0d1d2a9df13b393b7d191543accb92bd6853e7b7d8882ec"} err="failed to get container status \"8ac7f6d90952257be0d1d2a9df13b393b7d191543accb92bd6853e7b7d8882ec\": rpc error: code = NotFound desc = could not find container \"8ac7f6d90952257be0d1d2a9df13b393b7d191543accb92bd6853e7b7d8882ec\": container with ID starting with 8ac7f6d90952257be0d1d2a9df13b393b7d191543accb92bd6853e7b7d8882ec not found: ID does not exist"
Sep 16 12:22:27.146533 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:27.146507    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"]
Sep 16 12:22:27.150474 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:27.150451    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw"]
Sep 16 12:22:27.213410 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:27.213323    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3107a49-9748-4a44-820b-7ab2130aee6d" path="/var/lib/kubelet/pods/f3107a49-9748-4a44-820b-7ab2130aee6d/volumes"
Sep 16 12:22:29.919319 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:29.919270    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d6af8fa6-ddb6bb6d4-wmzgw" podUID="f3107a49-9748-4a44-820b-7ab2130aee6d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.61:8443/health\": context deadline exceeded"
Sep 16 12:22:34.528448 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:34.528408    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:34.528887 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:34.528501    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:34.541209 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:34.541184    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:35.178807 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:35.178779    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:22:52.417211 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.417170    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"]
Sep 16 12:22:52.417855 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.417835    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3107a49-9748-4a44-820b-7ab2130aee6d" containerName="maas-api"
Sep 16 12:22:52.417925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.417858    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3107a49-9748-4a44-820b-7ab2130aee6d" containerName="maas-api"
Sep 16 12:22:52.417969 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.417958    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3107a49-9748-4a44-820b-7ab2130aee6d" containerName="maas-api"
Sep 16 12:22:52.422539 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.422517    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.427191 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.426814    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9a49af\"/\"openshift-service-ca.crt\""
Sep 16 12:22:52.427191 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.426828    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9a49af\"/\"default-dockercfg-kszmt\""
Sep 16 12:22:52.427191 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.426965    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9a49af\"/\"test-model-9a49af-kserve-self-signed-certs\""
Sep 16 12:22:52.427191 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.427094    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9a49af\"/\"kube-root-ca.crt\""
Sep 16 12:22:52.430376 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.430006    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"]
Sep 16 12:22:52.533985 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.533948    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-home\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.534174 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.533998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stfks\" (UniqueName: \"kubernetes.io/projected/42829500-7641-45d5-956d-60e26554756e-kube-api-access-stfks\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.534174 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.534065    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-model-cache\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.534174 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.534106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-tmp-dir\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.534174 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.534157    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-dshm\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.534494 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.534223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42829500-7641-45d5-956d-60e26554756e-tls-certs\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.635334 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.635285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42829500-7641-45d5-956d-60e26554756e-tls-certs\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.635535 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.635407    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-home\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.635535 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.635457    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stfks\" (UniqueName: \"kubernetes.io/projected/42829500-7641-45d5-956d-60e26554756e-kube-api-access-stfks\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.635535 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.635502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-model-cache\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.635766 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.635544    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-tmp-dir\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.635766 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.635575    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-dshm\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.638775 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.638721    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-dshm\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.641897 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.641844    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42829500-7641-45d5-956d-60e26554756e-tls-certs\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.642078 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.641901    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-home\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.642078 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.642011    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-tmp-dir\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.642078 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.642051    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-model-cache\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.650829 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.650799    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stfks\" (UniqueName: \"kubernetes.io/projected/42829500-7641-45d5-956d-60e26554756e-kube-api-access-stfks\") pod \"test-model-9a49af-kserve-66d69598b9-6vrlj\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") " pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:52.733478 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:52.733380    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:22:53.008192 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:53.008165    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"]
Sep 16 12:22:53.009303 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:22:53.009278    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42829500_7641_45d5_956d_60e26554756e.slice/crio-fdf0e44a4140ef9c684a4532b2292087b0ad0645d4c39fef4ac98a9674931f75 WatchSource:0}: Error finding container fdf0e44a4140ef9c684a4532b2292087b0ad0645d4c39fef4ac98a9674931f75: Status 404 returned error can't find the container with id fdf0e44a4140ef9c684a4532b2292087b0ad0645d4c39fef4ac98a9674931f75
Sep 16 12:22:53.276642 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:53.276539    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj" event={"ID":"42829500-7641-45d5-956d-60e26554756e","Type":"ContainerStarted","Data":"a345c56b53281db61cd4b73cfa3421694302ea5995c4620a54794cf5ef232640"}
Sep 16 12:22:53.276642 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:53.276590    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj" event={"ID":"42829500-7641-45d5-956d-60e26554756e","Type":"ContainerStarted","Data":"fdf0e44a4140ef9c684a4532b2292087b0ad0645d4c39fef4ac98a9674931f75"}
Sep 16 12:22:53.293726 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:22:53.293666    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj" podStartSLOduration=1.293649796 podStartE2EDuration="1.293649796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:22:53.291834176 +0000 UTC m=+1332.542371716" watchObservedRunningTime="2026-09-16 12:22:53.293649796 +0000 UTC m=+1332.544187345"
Sep 16 12:23:02.734442 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:02.734339    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:23:02.734442 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:02.734397    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:23:02.746992 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:02.746967    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:23:03.346532 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:03.346494    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:23:07.467437 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:07.467398    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"]
Sep 16 12:23:07.468107 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:07.468059    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2" podUID="0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4" containerName="maas-api" containerID="cri-o://92577b7f6ca1b38aa68e807f08ab87b933557dec1d0cf42920df1f3197764c35" gracePeriod=30
Sep 16 12:23:07.717916 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:07.717845    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:23:07.774657 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:07.774621    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-maas-api-tls\") pod \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\" (UID: \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\") "
Sep 16 12:23:07.774877 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:07.774681    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-metrics-certs\") pod \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\" (UID: \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\") "
Sep 16 12:23:07.774877 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:07.774723    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s8hv\" (UniqueName: \"kubernetes.io/projected/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-kube-api-access-9s8hv\") pod \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\" (UID: \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\") "
Sep 16 12:23:07.774877 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:07.774816    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-openshift-service-ca\") pod \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\" (UID: \"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4\") "
Sep 16 12:23:07.775167 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:07.775141    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4" (UID: "0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:23:07.776925 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:07.776902    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4" (UID: "0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:23:07.777413 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:07.777394    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-kube-api-access-9s8hv" (OuterVolumeSpecName: "kube-api-access-9s8hv") pod "0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4" (UID: "0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4"). InnerVolumeSpecName "kube-api-access-9s8hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:23:07.777413 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:07.777398    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4" (UID: "0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:23:07.875816 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:07.875772    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:07.875816 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:07.875800    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s8hv\" (UniqueName: \"kubernetes.io/projected/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-kube-api-access-9s8hv\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:07.875816 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:07.875819    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:07.875816 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:07.875828    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:08.365278 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:08.365235    2582 generic.go:356] "Generic (PLEG): container finished" podID="0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4" containerID="92577b7f6ca1b38aa68e807f08ab87b933557dec1d0cf42920df1f3197764c35" exitCode=0
Sep 16 12:23:08.365476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:08.365294    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2" event={"ID":"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4","Type":"ContainerDied","Data":"92577b7f6ca1b38aa68e807f08ab87b933557dec1d0cf42920df1f3197764c35"}
Sep 16 12:23:08.365476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:08.365343    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2" event={"ID":"0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4","Type":"ContainerDied","Data":"0a21ed4ff7dfb97acb4f4f3631ed0fd571a4db6fb029fd20ad44c89ce2914c07"}
Sep 16 12:23:08.365476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:08.365366    2582 scope.go:117] "RemoveContainer" containerID="92577b7f6ca1b38aa68e807f08ab87b933557dec1d0cf42920df1f3197764c35"
Sep 16 12:23:08.365476 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:08.365296    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"
Sep 16 12:23:08.375274 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:08.375256    2582 scope.go:117] "RemoveContainer" containerID="92577b7f6ca1b38aa68e807f08ab87b933557dec1d0cf42920df1f3197764c35"
Sep 16 12:23:08.375527 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:23:08.375507    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92577b7f6ca1b38aa68e807f08ab87b933557dec1d0cf42920df1f3197764c35\": container with ID starting with 92577b7f6ca1b38aa68e807f08ab87b933557dec1d0cf42920df1f3197764c35 not found: ID does not exist" containerID="92577b7f6ca1b38aa68e807f08ab87b933557dec1d0cf42920df1f3197764c35"
Sep 16 12:23:08.375585 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:08.375536    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92577b7f6ca1b38aa68e807f08ab87b933557dec1d0cf42920df1f3197764c35"} err="failed to get container status \"92577b7f6ca1b38aa68e807f08ab87b933557dec1d0cf42920df1f3197764c35\": rpc error: code = NotFound desc = could not find container \"92577b7f6ca1b38aa68e807f08ab87b933557dec1d0cf42920df1f3197764c35\": container with ID starting with 92577b7f6ca1b38aa68e807f08ab87b933557dec1d0cf42920df1f3197764c35 not found: ID does not exist"
Sep 16 12:23:08.391504 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:08.391465    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"]
Sep 16 12:23:08.395102 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:08.395067    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7c3429-58d7c8d4-lc8n2"]
Sep 16 12:23:09.213108 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:09.213067    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4" path="/var/lib/kubelet/pods/0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4/volumes"
Sep 16 12:23:23.132013 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:23.131964    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"]
Sep 16 12:23:29.398728 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:29.398679    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"]
Sep 16 12:23:34.921272 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:34.921232    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"]
Sep 16 12:23:34.921835 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:34.921500    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v" podUID="c60d328f-e42b-46d0-b92c-dccf10b62790" containerName="maas-api" containerID="cri-o://bfca2629f843d7443b68b11637dfa256f32dd4222718d952d4eb84730781ed2c" gracePeriod=30
Sep 16 12:23:35.168099 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.168067    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:23:35.342178 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.342140    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c60d328f-e42b-46d0-b92c-dccf10b62790-openshift-service-ca\") pod \"c60d328f-e42b-46d0-b92c-dccf10b62790\" (UID: \"c60d328f-e42b-46d0-b92c-dccf10b62790\") "
Sep 16 12:23:35.342385 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.342218    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6xsf\" (UniqueName: \"kubernetes.io/projected/c60d328f-e42b-46d0-b92c-dccf10b62790-kube-api-access-z6xsf\") pod \"c60d328f-e42b-46d0-b92c-dccf10b62790\" (UID: \"c60d328f-e42b-46d0-b92c-dccf10b62790\") "
Sep 16 12:23:35.342385 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.342284    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c60d328f-e42b-46d0-b92c-dccf10b62790-metrics-certs\") pod \"c60d328f-e42b-46d0-b92c-dccf10b62790\" (UID: \"c60d328f-e42b-46d0-b92c-dccf10b62790\") "
Sep 16 12:23:35.342385 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.342300    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c60d328f-e42b-46d0-b92c-dccf10b62790-maas-api-tls\") pod \"c60d328f-e42b-46d0-b92c-dccf10b62790\" (UID: \"c60d328f-e42b-46d0-b92c-dccf10b62790\") "
Sep 16 12:23:35.342587 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.342561    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60d328f-e42b-46d0-b92c-dccf10b62790-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c60d328f-e42b-46d0-b92c-dccf10b62790" (UID: "c60d328f-e42b-46d0-b92c-dccf10b62790"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:23:35.344595 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.344566    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c60d328f-e42b-46d0-b92c-dccf10b62790-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c60d328f-e42b-46d0-b92c-dccf10b62790" (UID: "c60d328f-e42b-46d0-b92c-dccf10b62790"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:23:35.344706 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.344612    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60d328f-e42b-46d0-b92c-dccf10b62790-kube-api-access-z6xsf" (OuterVolumeSpecName: "kube-api-access-z6xsf") pod "c60d328f-e42b-46d0-b92c-dccf10b62790" (UID: "c60d328f-e42b-46d0-b92c-dccf10b62790"). InnerVolumeSpecName "kube-api-access-z6xsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:23:35.344706 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.344658    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c60d328f-e42b-46d0-b92c-dccf10b62790-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c60d328f-e42b-46d0-b92c-dccf10b62790" (UID: "c60d328f-e42b-46d0-b92c-dccf10b62790"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:23:35.443205 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.443173    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c60d328f-e42b-46d0-b92c-dccf10b62790-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:35.443205 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.443199    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c60d328f-e42b-46d0-b92c-dccf10b62790-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:35.443205 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.443208    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c60d328f-e42b-46d0-b92c-dccf10b62790-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:35.443441 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.443221    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6xsf\" (UniqueName: \"kubernetes.io/projected/c60d328f-e42b-46d0-b92c-dccf10b62790-kube-api-access-z6xsf\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:35.525387 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.525352    2582 generic.go:356] "Generic (PLEG): container finished" podID="c60d328f-e42b-46d0-b92c-dccf10b62790" containerID="bfca2629f843d7443b68b11637dfa256f32dd4222718d952d4eb84730781ed2c" exitCode=0
Sep 16 12:23:35.525565 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.525405    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"
Sep 16 12:23:35.525565 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.525444    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v" event={"ID":"c60d328f-e42b-46d0-b92c-dccf10b62790","Type":"ContainerDied","Data":"bfca2629f843d7443b68b11637dfa256f32dd4222718d952d4eb84730781ed2c"}
Sep 16 12:23:35.525565 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.525503    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v" event={"ID":"c60d328f-e42b-46d0-b92c-dccf10b62790","Type":"ContainerDied","Data":"bc864b5955ebd3bbcea17209f082f98a0c296dc03f347d6bb49099ca4e5fcba9"}
Sep 16 12:23:35.525565 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.525529    2582 scope.go:117] "RemoveContainer" containerID="bfca2629f843d7443b68b11637dfa256f32dd4222718d952d4eb84730781ed2c"
Sep 16 12:23:35.536164 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.536142    2582 scope.go:117] "RemoveContainer" containerID="bfca2629f843d7443b68b11637dfa256f32dd4222718d952d4eb84730781ed2c"
Sep 16 12:23:35.536494 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:23:35.536471    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfca2629f843d7443b68b11637dfa256f32dd4222718d952d4eb84730781ed2c\": container with ID starting with bfca2629f843d7443b68b11637dfa256f32dd4222718d952d4eb84730781ed2c not found: ID does not exist" containerID="bfca2629f843d7443b68b11637dfa256f32dd4222718d952d4eb84730781ed2c"
Sep 16 12:23:35.536569 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.536510    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfca2629f843d7443b68b11637dfa256f32dd4222718d952d4eb84730781ed2c"} err="failed to get container status \"bfca2629f843d7443b68b11637dfa256f32dd4222718d952d4eb84730781ed2c\": rpc error: code = NotFound desc = could not find container \"bfca2629f843d7443b68b11637dfa256f32dd4222718d952d4eb84730781ed2c\": container with ID starting with bfca2629f843d7443b68b11637dfa256f32dd4222718d952d4eb84730781ed2c not found: ID does not exist"
Sep 16 12:23:35.549206 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.549182    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"]
Sep 16 12:23:35.551418 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:35.551397    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0e8ffe-7cfd5c8b4f-b9n7v"]
Sep 16 12:23:37.212762 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:37.212694    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c60d328f-e42b-46d0-b92c-dccf10b62790" path="/var/lib/kubelet/pods/c60d328f-e42b-46d0-b92c-dccf10b62790/volumes"
Sep 16 12:23:38.152501 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.152460    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk" podUID="5692b6cf-69ba-4249-9ec2-a38a7ba45372" containerName="main" containerID="cri-o://af324ed7b0b92cdaf35da26638d5300b4274d127cd50bc71d98ed9cb2c1341eb" gracePeriod=45
Sep 16 12:23:38.412851 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.412783    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:23:38.544808 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.544778    2582 generic.go:356] "Generic (PLEG): container finished" podID="5692b6cf-69ba-4249-9ec2-a38a7ba45372" containerID="af324ed7b0b92cdaf35da26638d5300b4274d127cd50bc71d98ed9cb2c1341eb" exitCode=0
Sep 16 12:23:38.544992 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.544827    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"
Sep 16 12:23:38.544992 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.544876    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk" event={"ID":"5692b6cf-69ba-4249-9ec2-a38a7ba45372","Type":"ContainerDied","Data":"af324ed7b0b92cdaf35da26638d5300b4274d127cd50bc71d98ed9cb2c1341eb"}
Sep 16 12:23:38.544992 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.544921    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk" event={"ID":"5692b6cf-69ba-4249-9ec2-a38a7ba45372","Type":"ContainerDied","Data":"28975a622cc65db28cc67e5f1d51754ae692b5e110aca75d678d000ca33c56f6"}
Sep 16 12:23:38.544992 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.544942    2582 scope.go:117] "RemoveContainer" containerID="af324ed7b0b92cdaf35da26638d5300b4274d127cd50bc71d98ed9cb2c1341eb"
Sep 16 12:23:38.554082 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.554060    2582 scope.go:117] "RemoveContainer" containerID="af324ed7b0b92cdaf35da26638d5300b4274d127cd50bc71d98ed9cb2c1341eb"
Sep 16 12:23:38.554395 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:23:38.554375    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af324ed7b0b92cdaf35da26638d5300b4274d127cd50bc71d98ed9cb2c1341eb\": container with ID starting with af324ed7b0b92cdaf35da26638d5300b4274d127cd50bc71d98ed9cb2c1341eb not found: ID does not exist" containerID="af324ed7b0b92cdaf35da26638d5300b4274d127cd50bc71d98ed9cb2c1341eb"
Sep 16 12:23:38.554460 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.554406    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af324ed7b0b92cdaf35da26638d5300b4274d127cd50bc71d98ed9cb2c1341eb"} err="failed to get container status \"af324ed7b0b92cdaf35da26638d5300b4274d127cd50bc71d98ed9cb2c1341eb\": rpc error: code = NotFound desc = could not find container \"af324ed7b0b92cdaf35da26638d5300b4274d127cd50bc71d98ed9cb2c1341eb\": container with ID starting with af324ed7b0b92cdaf35da26638d5300b4274d127cd50bc71d98ed9cb2c1341eb not found: ID does not exist"
Sep 16 12:23:38.572774 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.572721    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5692b6cf-69ba-4249-9ec2-a38a7ba45372-tls-certs\") pod \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") "
Sep 16 12:23:38.572914 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.572807    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-model-cache\") pod \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") "
Sep 16 12:23:38.572914 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.572837    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-home\") pod \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") "
Sep 16 12:23:38.572914 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.572874    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-dshm\") pod \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") "
Sep 16 12:23:38.573093 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.572926    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-tmp-dir\") pod \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") "
Sep 16 12:23:38.573093 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.572960    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfwv5\" (UniqueName: \"kubernetes.io/projected/5692b6cf-69ba-4249-9ec2-a38a7ba45372-kube-api-access-hfwv5\") pod \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\" (UID: \"5692b6cf-69ba-4249-9ec2-a38a7ba45372\") "
Sep 16 12:23:38.573093 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.573074    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-model-cache" (OuterVolumeSpecName: "model-cache") pod "5692b6cf-69ba-4249-9ec2-a38a7ba45372" (UID: "5692b6cf-69ba-4249-9ec2-a38a7ba45372"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:23:38.573246 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.573129    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-home" (OuterVolumeSpecName: "home") pod "5692b6cf-69ba-4249-9ec2-a38a7ba45372" (UID: "5692b6cf-69ba-4249-9ec2-a38a7ba45372"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:23:38.573246 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.573167    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5692b6cf-69ba-4249-9ec2-a38a7ba45372" (UID: "5692b6cf-69ba-4249-9ec2-a38a7ba45372"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:23:38.573333 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.573292    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:38.573333 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.573303    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:38.573333 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.573317    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:38.575129 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.575102    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5692b6cf-69ba-4249-9ec2-a38a7ba45372-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5692b6cf-69ba-4249-9ec2-a38a7ba45372" (UID: "5692b6cf-69ba-4249-9ec2-a38a7ba45372"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:23:38.575227 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.575149    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5692b6cf-69ba-4249-9ec2-a38a7ba45372-kube-api-access-hfwv5" (OuterVolumeSpecName: "kube-api-access-hfwv5") pod "5692b6cf-69ba-4249-9ec2-a38a7ba45372" (UID: "5692b6cf-69ba-4249-9ec2-a38a7ba45372"). InnerVolumeSpecName "kube-api-access-hfwv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:23:38.575227 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.575179    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-dshm" (OuterVolumeSpecName: "dshm") pod "5692b6cf-69ba-4249-9ec2-a38a7ba45372" (UID: "5692b6cf-69ba-4249-9ec2-a38a7ba45372"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:23:38.674126 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.674043    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfwv5\" (UniqueName: \"kubernetes.io/projected/5692b6cf-69ba-4249-9ec2-a38a7ba45372-kube-api-access-hfwv5\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:38.674126 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.674071    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5692b6cf-69ba-4249-9ec2-a38a7ba45372-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:38.674126 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.674081    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5692b6cf-69ba-4249-9ec2-a38a7ba45372-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:38.866462 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.866432    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"]
Sep 16 12:23:38.872135 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:38.872101    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-7c3429/autores-model-7c3429-kserve-5c6b89499f-hxvhk"]
Sep 16 12:23:39.218331 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:39.218294    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5692b6cf-69ba-4249-9ec2-a38a7ba45372" path="/var/lib/kubelet/pods/5692b6cf-69ba-4249-9ec2-a38a7ba45372/volumes"
Sep 16 12:23:44.419711 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.419667    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx" podUID="c685a655-d0c4-4f1d-87d0-130974c523fd" containerName="main" containerID="cri-o://07161ce81535fd8432e6f3e8cc3b1544211886bb36577ebf5f1deda6de625147" gracePeriod=45
Sep 16 12:23:44.580030 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.579998    2582 generic.go:356] "Generic (PLEG): container finished" podID="c685a655-d0c4-4f1d-87d0-130974c523fd" containerID="07161ce81535fd8432e6f3e8cc3b1544211886bb36577ebf5f1deda6de625147" exitCode=0
Sep 16 12:23:44.580194 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.580068    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx" event={"ID":"c685a655-d0c4-4f1d-87d0-130974c523fd","Type":"ContainerDied","Data":"07161ce81535fd8432e6f3e8cc3b1544211886bb36577ebf5f1deda6de625147"}
Sep 16 12:23:44.675337 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.675278    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:23:44.720047 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.720011    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-home\") pod \"c685a655-d0c4-4f1d-87d0-130974c523fd\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") "
Sep 16 12:23:44.720272 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.720093    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvd9m\" (UniqueName: \"kubernetes.io/projected/c685a655-d0c4-4f1d-87d0-130974c523fd-kube-api-access-jvd9m\") pod \"c685a655-d0c4-4f1d-87d0-130974c523fd\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") "
Sep 16 12:23:44.720272 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.720146    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-dshm\") pod \"c685a655-d0c4-4f1d-87d0-130974c523fd\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") "
Sep 16 12:23:44.720272 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.720196    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c685a655-d0c4-4f1d-87d0-130974c523fd-tls-certs\") pod \"c685a655-d0c4-4f1d-87d0-130974c523fd\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") "
Sep 16 12:23:44.720272 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.720232    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-model-cache\") pod \"c685a655-d0c4-4f1d-87d0-130974c523fd\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") "
Sep 16 12:23:44.720272 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.720259    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-tmp-dir\") pod \"c685a655-d0c4-4f1d-87d0-130974c523fd\" (UID: \"c685a655-d0c4-4f1d-87d0-130974c523fd\") "
Sep 16 12:23:44.720585 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.720283    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-home" (OuterVolumeSpecName: "home") pod "c685a655-d0c4-4f1d-87d0-130974c523fd" (UID: "c685a655-d0c4-4f1d-87d0-130974c523fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:23:44.720652 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.720591    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:44.720848 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.720815    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c685a655-d0c4-4f1d-87d0-130974c523fd" (UID: "c685a655-d0c4-4f1d-87d0-130974c523fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:23:44.720956 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.720832    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "c685a655-d0c4-4f1d-87d0-130974c523fd" (UID: "c685a655-d0c4-4f1d-87d0-130974c523fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:23:44.722783 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.722724    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c685a655-d0c4-4f1d-87d0-130974c523fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c685a655-d0c4-4f1d-87d0-130974c523fd" (UID: "c685a655-d0c4-4f1d-87d0-130974c523fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:23:44.722918 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.722900    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-dshm" (OuterVolumeSpecName: "dshm") pod "c685a655-d0c4-4f1d-87d0-130974c523fd" (UID: "c685a655-d0c4-4f1d-87d0-130974c523fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:23:44.723008 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.722981    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c685a655-d0c4-4f1d-87d0-130974c523fd-kube-api-access-jvd9m" (OuterVolumeSpecName: "kube-api-access-jvd9m") pod "c685a655-d0c4-4f1d-87d0-130974c523fd" (UID: "c685a655-d0c4-4f1d-87d0-130974c523fd"). InnerVolumeSpecName "kube-api-access-jvd9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:23:44.821804 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.821726    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvd9m\" (UniqueName: \"kubernetes.io/projected/c685a655-d0c4-4f1d-87d0-130974c523fd-kube-api-access-jvd9m\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:44.821804 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.821799    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:44.821804 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.821810    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c685a655-d0c4-4f1d-87d0-130974c523fd-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:44.822052 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.821820    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:44.822052 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:44.821828    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c685a655-d0c4-4f1d-87d0-130974c523fd-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:23:45.587437 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:45.587409    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"
Sep 16 12:23:45.587896 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:45.587448    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx" event={"ID":"c685a655-d0c4-4f1d-87d0-130974c523fd","Type":"ContainerDied","Data":"8f1a5010ccfd0bd21458d7fd846e08b2905fc8e1a5c0ef5ed6caf2354c0389b0"}
Sep 16 12:23:45.587896 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:45.587537    2582 scope.go:117] "RemoveContainer" containerID="07161ce81535fd8432e6f3e8cc3b1544211886bb36577ebf5f1deda6de625147"
Sep 16 12:23:45.604311 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:45.604282    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"]
Sep 16 12:23:45.608246 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:45.608215    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0e8ffe/simulator-w0-0e8ffe-kserve-695b5bb49-99gvx"]
Sep 16 12:23:47.212595 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:47.212560    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c685a655-d0c4-4f1d-87d0-130974c523fd" path="/var/lib/kubelet/pods/c685a655-d0c4-4f1d-87d0-130974c523fd/volumes"
Sep 16 12:23:51.816528 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.816485    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"]
Sep 16 12:23:51.817094 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.817074    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4" containerName="maas-api"
Sep 16 12:23:51.817158 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.817099    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4" containerName="maas-api"
Sep 16 12:23:51.817158 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.817116    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c685a655-d0c4-4f1d-87d0-130974c523fd" containerName="main"
Sep 16 12:23:51.817158 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.817125    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c685a655-d0c4-4f1d-87d0-130974c523fd" containerName="main"
Sep 16 12:23:51.817158 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.817134    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5692b6cf-69ba-4249-9ec2-a38a7ba45372" containerName="main"
Sep 16 12:23:51.817158 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.817141    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5692b6cf-69ba-4249-9ec2-a38a7ba45372" containerName="main"
Sep 16 12:23:51.817158 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.817154    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c60d328f-e42b-46d0-b92c-dccf10b62790" containerName="maas-api"
Sep 16 12:23:51.817346 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.817162    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c60d328f-e42b-46d0-b92c-dccf10b62790" containerName="maas-api"
Sep 16 12:23:51.817346 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.817253    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d3f3a7c-b235-4f34-83a1-c3e8b8fdcbe4" containerName="maas-api"
Sep 16 12:23:51.817346 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.817266    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5692b6cf-69ba-4249-9ec2-a38a7ba45372" containerName="main"
Sep 16 12:23:51.817346 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.817275    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c60d328f-e42b-46d0-b92c-dccf10b62790" containerName="maas-api"
Sep 16 12:23:51.817346 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.817285    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c685a655-d0c4-4f1d-87d0-130974c523fd" containerName="main"
Sep 16 12:23:51.820904 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.820877    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:51.823969 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.823943    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7a4d5c\"/\"premium-w4-7a4d5c-kserve-self-signed-certs\""
Sep 16 12:23:51.832631 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.832604    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"]
Sep 16 12:23:51.890086 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.890039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-dshm\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:51.890086 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.890091    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-home\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:51.890319 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.890159    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a3df43-e996-42d0-93e2-d13b60da875c-tls-certs\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:51.890319 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.890281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-model-cache\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:51.890319 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.890307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-tmp-dir\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:51.890414 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.890336    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lt29\" (UniqueName: \"kubernetes.io/projected/d1a3df43-e996-42d0-93e2-d13b60da875c-kube-api-access-7lt29\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:51.991840 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.991798    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-model-cache\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:51.991840 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.991846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-tmp-dir\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:51.992102 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.991974    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lt29\" (UniqueName: \"kubernetes.io/projected/d1a3df43-e996-42d0-93e2-d13b60da875c-kube-api-access-7lt29\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:51.992102 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.992036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-dshm\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:51.992102 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.992069    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-home\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:51.992264 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.992142    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a3df43-e996-42d0-93e2-d13b60da875c-tls-certs\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:51.992264 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.992238    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-model-cache\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:51.992264 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.992273    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-tmp-dir\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:51.992444 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.992425    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-home\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:51.994378 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.994357    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-dshm\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:51.994695 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:51.994680    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a3df43-e996-42d0-93e2-d13b60da875c-tls-certs\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:52.000039 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:52.000015    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lt29\" (UniqueName: \"kubernetes.io/projected/d1a3df43-e996-42d0-93e2-d13b60da875c-kube-api-access-7lt29\") pod \"premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") " pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:52.133424 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:52.133380    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:23:52.405723 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:52.405693    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"]
Sep 16 12:23:52.406937 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:23:52.406908    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1a3df43_e996_42d0_93e2_d13b60da875c.slice/crio-ebd9f089dd230a9668bd4d693ccc42a34954a6847c905bdb82c69f51d30d1e1f WatchSource:0}: Error finding container ebd9f089dd230a9668bd4d693ccc42a34954a6847c905bdb82c69f51d30d1e1f: Status 404 returned error can't find the container with id ebd9f089dd230a9668bd4d693ccc42a34954a6847c905bdb82c69f51d30d1e1f
Sep 16 12:23:52.630829 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:52.630791    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s" event={"ID":"d1a3df43-e996-42d0-93e2-d13b60da875c","Type":"ContainerStarted","Data":"ffef68f480536e8dd79ccfef41100f8329fab94e6959446340a6123ea93febca"}
Sep 16 12:23:52.631002 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:52.630837    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s" event={"ID":"d1a3df43-e996-42d0-93e2-d13b60da875c","Type":"ContainerStarted","Data":"ebd9f089dd230a9668bd4d693ccc42a34954a6847c905bdb82c69f51d30d1e1f"}
Sep 16 12:23:52.646330 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:52.646279    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s" podStartSLOduration=1.6462629149999999 podStartE2EDuration="1.646262915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:23:52.645790368 +0000 UTC m=+1391.896327918" watchObservedRunningTime="2026-09-16 12:23:52.646262915 +0000 UTC m=+1391.896800462"
Sep 16 12:23:58.202859 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.202824    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"]
Sep 16 12:23:58.207393 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.207359    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.212169 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.212136    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4bef4d\"/\"test-model-4bef4d-kserve-self-signed-certs\""
Sep 16 12:23:58.212454 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.212433    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4bef4d\"/\"kube-root-ca.crt\""
Sep 16 12:23:58.212699 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.212679    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4bef4d\"/\"openshift-service-ca.crt\""
Sep 16 12:23:58.213765 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.213717    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4bef4d\"/\"default-dockercfg-dfjt6\""
Sep 16 12:23:58.217420 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.217391    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"]
Sep 16 12:23:58.254521 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.254490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-dshm\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.254521 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.254527    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-home\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.254728 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.254565    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a9dfed-3f79-4f97-83a1-93f376fd302f-tls-certs\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.254728 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.254620    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-model-cache\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.254728 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.254636    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk2xw\" (UniqueName: \"kubernetes.io/projected/a5a9dfed-3f79-4f97-83a1-93f376fd302f-kube-api-access-rk2xw\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.254728 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.254674    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-tmp-dir\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.355844 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.355792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-dshm\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.355844 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.355848    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-home\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.356090 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.355874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a9dfed-3f79-4f97-83a1-93f376fd302f-tls-certs\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.356090 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.355916    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-model-cache\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.356090 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.355933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk2xw\" (UniqueName: \"kubernetes.io/projected/a5a9dfed-3f79-4f97-83a1-93f376fd302f-kube-api-access-rk2xw\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.356090 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.355954    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-tmp-dir\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.356316 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.356290    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-home\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.356380 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.356301    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-model-cache\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.356426 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.356387    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-tmp-dir\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.358571 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.358551    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a9dfed-3f79-4f97-83a1-93f376fd302f-tls-certs\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.358668 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.358636    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-dshm\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.366725 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.366694    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk2xw\" (UniqueName: \"kubernetes.io/projected/a5a9dfed-3f79-4f97-83a1-93f376fd302f-kube-api-access-rk2xw\") pod \"test-model-4bef4d-kserve-7f54468f55-zs7k5\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") " pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.522308 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.522213    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:23:58.793752 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:58.793690    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"]
Sep 16 12:23:58.795280 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:23:58.795252    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5a9dfed_3f79_4f97_83a1_93f376fd302f.slice/crio-d4a87b882dbcbfd01776b7ae907d83d589eeb6c185db0c765a12cf48aa8e6973 WatchSource:0}: Error finding container d4a87b882dbcbfd01776b7ae907d83d589eeb6c185db0c765a12cf48aa8e6973: Status 404 returned error can't find the container with id d4a87b882dbcbfd01776b7ae907d83d589eeb6c185db0c765a12cf48aa8e6973
Sep 16 12:23:59.673059 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:59.673016    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5" event={"ID":"a5a9dfed-3f79-4f97-83a1-93f376fd302f","Type":"ContainerStarted","Data":"f3a14b7d6b113b2988d8996dedded5197026c05b73f9c7d59222b1348e9dd58b"}
Sep 16 12:23:59.673059 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:59.673064    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5" event={"ID":"a5a9dfed-3f79-4f97-83a1-93f376fd302f","Type":"ContainerStarted","Data":"d4a87b882dbcbfd01776b7ae907d83d589eeb6c185db0c765a12cf48aa8e6973"}
Sep 16 12:23:59.691268 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:23:59.691217    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5" podStartSLOduration=1.691200611 podStartE2EDuration="1.691200611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:23:59.689549153 +0000 UTC m=+1398.940086704" watchObservedRunningTime="2026-09-16 12:23:59.691200611 +0000 UTC m=+1398.941738207"
Sep 16 12:24:02.134329 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:02.134283    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:24:02.134809 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:02.134344    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:24:02.147214 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:02.147186    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:24:02.701034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:02.701002    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:24:06.221111 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:06.221068    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"]
Sep 16 12:24:06.225110 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:06.225084    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:24:06.228096 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:06.228065    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-4915f8-serving-cert\""
Sep 16 12:24:06.237843 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:06.236361    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"]
Sep 16 12:24:06.331691 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:06.331650    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adc4128a-7b81-45f1-8682-cb2351c9f9ee-maas-api-tls\") pod \"maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk\" (UID: \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:24:06.331921 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:06.331800    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adc4128a-7b81-45f1-8682-cb2351c9f9ee-metrics-certs\") pod \"maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk\" (UID: \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:24:06.331921 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:06.331856    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh9b9\" (UniqueName: \"kubernetes.io/projected/adc4128a-7b81-45f1-8682-cb2351c9f9ee-kube-api-access-jh9b9\") pod \"maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk\" (UID: \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:24:06.332047 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:06.332005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc4128a-7b81-45f1-8682-cb2351c9f9ee-openshift-service-ca\") pod \"maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk\" (UID: \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:24:06.433418 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:06.433371    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc4128a-7b81-45f1-8682-cb2351c9f9ee-openshift-service-ca\") pod \"maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk\" (UID: \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:24:06.433591 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:06.433467    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adc4128a-7b81-45f1-8682-cb2351c9f9ee-maas-api-tls\") pod \"maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk\" (UID: \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:24:06.433591 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:06.433543    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adc4128a-7b81-45f1-8682-cb2351c9f9ee-metrics-certs\") pod \"maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk\" (UID: \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:24:06.433665 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:06.433591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh9b9\" (UniqueName: \"kubernetes.io/projected/adc4128a-7b81-45f1-8682-cb2351c9f9ee-kube-api-access-jh9b9\") pod \"maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk\" (UID: \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:24:06.433665 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:24:06.433609    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-serving-cert: secret "maas-api-e2e-explicit-4915f8-serving-cert" not found
Sep 16 12:24:06.433757 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:24:06.433692    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adc4128a-7b81-45f1-8682-cb2351c9f9ee-maas-api-tls podName:adc4128a-7b81-45f1-8682-cb2351c9f9ee nodeName:}" failed. No retries permitted until 2026-09-16 12:24:06.933673481 +0000 UTC m=+1406.184211008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/adc4128a-7b81-45f1-8682-cb2351c9f9ee-maas-api-tls") pod "maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk" (UID: "adc4128a-7b81-45f1-8682-cb2351c9f9ee") : secret "maas-api-e2e-explicit-4915f8-serving-cert" not found
Sep 16 12:24:06.434151 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:06.434130    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc4128a-7b81-45f1-8682-cb2351c9f9ee-openshift-service-ca\") pod \"maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk\" (UID: \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:24:06.436347 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:06.436278    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adc4128a-7b81-45f1-8682-cb2351c9f9ee-metrics-certs\") pod \"maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk\" (UID: \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:24:06.443956 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:06.443928    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh9b9\" (UniqueName: \"kubernetes.io/projected/adc4128a-7b81-45f1-8682-cb2351c9f9ee-kube-api-access-jh9b9\") pod \"maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk\" (UID: \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:24:06.938830 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:06.938791    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adc4128a-7b81-45f1-8682-cb2351c9f9ee-maas-api-tls\") pod \"maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk\" (UID: \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:24:06.941432 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:06.941406    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adc4128a-7b81-45f1-8682-cb2351c9f9ee-maas-api-tls\") pod \"maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk\" (UID: \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:24:07.140533 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:07.140495    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:24:07.416034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:07.415668    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"]
Sep 16 12:24:07.418642 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:24:07.418612    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc4128a_7b81_45f1_8682_cb2351c9f9ee.slice/crio-335ac9267ac34ec0fa5589afba1fa4c4492744ca0d5efb2b5347b55f21ea8885 WatchSource:0}: Error finding container 335ac9267ac34ec0fa5589afba1fa4c4492744ca0d5efb2b5347b55f21ea8885: Status 404 returned error can't find the container with id 335ac9267ac34ec0fa5589afba1fa4c4492744ca0d5efb2b5347b55f21ea8885
Sep 16 12:24:07.420167 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:07.420147    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:24:07.720314 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:07.720275    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk" event={"ID":"adc4128a-7b81-45f1-8682-cb2351c9f9ee","Type":"ContainerStarted","Data":"335ac9267ac34ec0fa5589afba1fa4c4492744ca0d5efb2b5347b55f21ea8885"}
Sep 16 12:24:08.523178 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:08.523139    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:24:08.523604 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:08.523199    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:24:08.535658 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:08.535633    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:24:08.727579 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:08.727544    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk" event={"ID":"adc4128a-7b81-45f1-8682-cb2351c9f9ee","Type":"ContainerStarted","Data":"09b1bb07e1640fe829c03003f5d85712a387dbb7c8561e68ceb1e866d905c25e"}
Sep 16 12:24:08.727795 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:08.727625    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:24:08.738996 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:08.738974    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:24:08.747720 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:08.747668    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk" podStartSLOduration=2.207579276 podStartE2EDuration="2.747648711s" totalImagesPullingTime="540.069435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:24:08.746847857 +0000 UTC m=+1407.997385407" watchObservedRunningTime="2026-09-16 12:24:08.747648711 +0000 UTC m=+1407.998186263"
Sep 16 12:24:14.739182 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:14.739147    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:24:32.826455 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:32.826360    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"]
Sep 16 12:24:32.831233 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:32.831208    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:32.834104 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:32.834078    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f2bc0450-serving-cert\""
Sep 16 12:24:32.841045 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:32.841017    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"]
Sep 16 12:24:32.886817 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:32.886780    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-metrics-certs\") pod \"maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:32.886817 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:32.886828    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-openshift-service-ca\") pod \"maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:32.887101 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:32.886855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-maas-api-tls\") pod \"maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:32.887101 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:32.886906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjxnf\" (UniqueName: \"kubernetes.io/projected/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-kube-api-access-xjxnf\") pod \"maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:32.988208 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:32.988170    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-metrics-certs\") pod \"maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:32.988208 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:32.988218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-openshift-service-ca\") pod \"maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:32.988451 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:32.988239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-maas-api-tls\") pod \"maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:32.988451 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:24:32.988327    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-serving-cert: secret "maas-api-e2e-mt-f2bc0450-serving-cert" not found
Sep 16 12:24:32.988451 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:32.988365    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjxnf\" (UniqueName: \"kubernetes.io/projected/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-kube-api-access-xjxnf\") pod \"maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:32.988451 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:24:32.988400    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-maas-api-tls podName:47ad3acd-7f77-4703-b3be-73eb51c1c1fb nodeName:}" failed. No retries permitted until 2026-09-16 12:24:33.488381712 +0000 UTC m=+1432.738919239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-maas-api-tls") pod "maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt" (UID: "47ad3acd-7f77-4703-b3be-73eb51c1c1fb") : secret "maas-api-e2e-mt-f2bc0450-serving-cert" not found
Sep 16 12:24:32.989034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:32.989007    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-openshift-service-ca\") pod \"maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:32.990998 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:32.990967    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-metrics-certs\") pod \"maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:33.001910 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:33.001886    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjxnf\" (UniqueName: \"kubernetes.io/projected/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-kube-api-access-xjxnf\") pod \"maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:33.495822 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:33.495776    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-maas-api-tls\") pod \"maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:33.496010 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:24:33.495933    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-serving-cert: secret "maas-api-e2e-mt-f2bc0450-serving-cert" not found
Sep 16 12:24:33.496057 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:24:33.496024    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-maas-api-tls podName:47ad3acd-7f77-4703-b3be-73eb51c1c1fb nodeName:}" failed. No retries permitted until 2026-09-16 12:24:34.496005717 +0000 UTC m=+1433.746543244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-maas-api-tls") pod "maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt" (UID: "47ad3acd-7f77-4703-b3be-73eb51c1c1fb") : secret "maas-api-e2e-mt-f2bc0450-serving-cert" not found
Sep 16 12:24:34.505101 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:34.505063    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-maas-api-tls\") pod \"maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:34.507997 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:34.507971    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-maas-api-tls\") pod \"maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:34.644079 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:34.644040    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:34.920430 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:34.920403    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"]
Sep 16 12:24:34.921572 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:24:34.921543    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ad3acd_7f77_4703_b3be_73eb51c1c1fb.slice/crio-92aee441df391bf3c789b010f499ae0b8f713aa379fb6bc459c436328e8fa9a5 WatchSource:0}: Error finding container 92aee441df391bf3c789b010f499ae0b8f713aa379fb6bc459c436328e8fa9a5: Status 404 returned error can't find the container with id 92aee441df391bf3c789b010f499ae0b8f713aa379fb6bc459c436328e8fa9a5
Sep 16 12:24:35.888758 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:35.888691    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt" event={"ID":"47ad3acd-7f77-4703-b3be-73eb51c1c1fb","Type":"ContainerStarted","Data":"db59890412f7d61c03fffc000740e0241bcc4b0fe748fc357addf21a51e8fc7d"}
Sep 16 12:24:35.889198 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:35.888810    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt" event={"ID":"47ad3acd-7f77-4703-b3be-73eb51c1c1fb","Type":"ContainerStarted","Data":"92aee441df391bf3c789b010f499ae0b8f713aa379fb6bc459c436328e8fa9a5"}
Sep 16 12:24:35.889198 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:35.888846    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:35.906283 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:35.906231    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt" podStartSLOduration=3.53253044 podStartE2EDuration="3.906214207s" totalImagesPullingTime="373.683767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:24:35.905523615 +0000 UTC m=+1435.156061167" watchObservedRunningTime="2026-09-16 12:24:35.906214207 +0000 UTC m=+1435.156751804"
Sep 16 12:24:41.900930 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:41.900899    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:24:59.907228 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:59.907183    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"]
Sep 16 12:24:59.912890 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:59.912859    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:24:59.915846 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:59.915820    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4915f8\"/\"explicit-model-4915f8-kserve-self-signed-certs\""
Sep 16 12:24:59.916029 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:59.915820    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4915f8\"/\"kube-root-ca.crt\""
Sep 16 12:24:59.916029 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:59.915820    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4915f8\"/\"openshift-service-ca.crt\""
Sep 16 12:24:59.916029 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:59.915827    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4915f8\"/\"default-dockercfg-lxs9w\""
Sep 16 12:24:59.922994 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:24:59.922962    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"]
Sep 16 12:25:00.052851 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.052820    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-home\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.053037 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.052861    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-model-cache\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.053037 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.052926    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f46s5\" (UniqueName: \"kubernetes.io/projected/05d4c2e8-cf18-4221-b418-5b57bfa519a3-kube-api-access-f46s5\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.053129 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.053060    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-dshm\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.053129 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.053091    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-tmp-dir\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.053129 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.053124    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05d4c2e8-cf18-4221-b418-5b57bfa519a3-tls-certs\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.154039 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.154002    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-dshm\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.154039 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.154044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-tmp-dir\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.154299 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.154066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05d4c2e8-cf18-4221-b418-5b57bfa519a3-tls-certs\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.154299 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.154107    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-home\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.154299 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.154127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-model-cache\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.154299 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.154153    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f46s5\" (UniqueName: \"kubernetes.io/projected/05d4c2e8-cf18-4221-b418-5b57bfa519a3-kube-api-access-f46s5\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.154523 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.154471    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-tmp-dir\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.154523 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.154495    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-model-cache\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.154632 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.154542    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-home\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.156601 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.156575    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-dshm\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.156906 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.156885    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05d4c2e8-cf18-4221-b418-5b57bfa519a3-tls-certs\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.162456 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.162400    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f46s5\" (UniqueName: \"kubernetes.io/projected/05d4c2e8-cf18-4221-b418-5b57bfa519a3-kube-api-access-f46s5\") pod \"explicit-model-4915f8-kserve-7fd978b59f-wt2d4\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") " pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.225067 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.225032    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:00.496929 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:00.496901    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"]
Sep 16 12:25:00.498387 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:25:00.498362    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05d4c2e8_cf18_4221_b418_5b57bfa519a3.slice/crio-d457458ddc3d6fc6ff192396bbbadf42d5a0bf3742b58ea701dd2fa4a63f6a2f WatchSource:0}: Error finding container d457458ddc3d6fc6ff192396bbbadf42d5a0bf3742b58ea701dd2fa4a63f6a2f: Status 404 returned error can't find the container with id d457458ddc3d6fc6ff192396bbbadf42d5a0bf3742b58ea701dd2fa4a63f6a2f
Sep 16 12:25:01.038263 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:01.038227    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4" event={"ID":"05d4c2e8-cf18-4221-b418-5b57bfa519a3","Type":"ContainerStarted","Data":"b6a5a3ed1aa0a5c0094b356351abd5f770938625037ec16a07c6c011945ecd74"}
Sep 16 12:25:01.038692 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:01.038275    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4" event={"ID":"05d4c2e8-cf18-4221-b418-5b57bfa519a3","Type":"ContainerStarted","Data":"d457458ddc3d6fc6ff192396bbbadf42d5a0bf3742b58ea701dd2fa4a63f6a2f"}
Sep 16 12:25:01.057706 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:01.057649    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4" podStartSLOduration=2.057633856 podStartE2EDuration="2.057633856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:25:01.055795701 +0000 UTC m=+1460.306333247" watchObservedRunningTime="2026-09-16 12:25:01.057633856 +0000 UTC m=+1460.308171404"
Sep 16 12:25:10.226174 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:10.226126    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:10.226685 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:10.226243    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:10.238802 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:10.238776    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:11.105056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:11.105029    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:13.520064 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:13.520026    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"]
Sep 16 12:25:13.520609 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:13.520284    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt" podUID="47ad3acd-7f77-4703-b3be-73eb51c1c1fb" containerName="maas-api" containerID="cri-o://db59890412f7d61c03fffc000740e0241bcc4b0fe748fc357addf21a51e8fc7d" gracePeriod=30
Sep 16 12:25:13.777526 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:13.777461    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:25:13.778459 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:13.778437    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-metrics-certs\") pod \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") "
Sep 16 12:25:13.778591 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:13.778491    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjxnf\" (UniqueName: \"kubernetes.io/projected/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-kube-api-access-xjxnf\") pod \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") "
Sep 16 12:25:13.778591 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:13.778522    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-maas-api-tls\") pod \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") "
Sep 16 12:25:13.778591 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:13.778558    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-openshift-service-ca\") pod \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\" (UID: \"47ad3acd-7f77-4703-b3be-73eb51c1c1fb\") "
Sep 16 12:25:13.778975 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:13.778949    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "47ad3acd-7f77-4703-b3be-73eb51c1c1fb" (UID: "47ad3acd-7f77-4703-b3be-73eb51c1c1fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:25:13.780605 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:13.780582    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "47ad3acd-7f77-4703-b3be-73eb51c1c1fb" (UID: "47ad3acd-7f77-4703-b3be-73eb51c1c1fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:25:13.780835 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:13.780812    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "47ad3acd-7f77-4703-b3be-73eb51c1c1fb" (UID: "47ad3acd-7f77-4703-b3be-73eb51c1c1fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:25:13.780835 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:13.780827    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-kube-api-access-xjxnf" (OuterVolumeSpecName: "kube-api-access-xjxnf") pod "47ad3acd-7f77-4703-b3be-73eb51c1c1fb" (UID: "47ad3acd-7f77-4703-b3be-73eb51c1c1fb"). InnerVolumeSpecName "kube-api-access-xjxnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:25:13.880275 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:13.880230    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjxnf\" (UniqueName: \"kubernetes.io/projected/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-kube-api-access-xjxnf\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:13.880275 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:13.880272    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:13.880472 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:13.880287    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:13.880472 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:13.880301    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ad3acd-7f77-4703-b3be-73eb51c1c1fb-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:14.112430 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:14.112391    2582 generic.go:356] "Generic (PLEG): container finished" podID="47ad3acd-7f77-4703-b3be-73eb51c1c1fb" containerID="db59890412f7d61c03fffc000740e0241bcc4b0fe748fc357addf21a51e8fc7d" exitCode=0
Sep 16 12:25:14.112631 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:14.112440    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"
Sep 16 12:25:14.112631 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:14.112478    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt" event={"ID":"47ad3acd-7f77-4703-b3be-73eb51c1c1fb","Type":"ContainerDied","Data":"db59890412f7d61c03fffc000740e0241bcc4b0fe748fc357addf21a51e8fc7d"}
Sep 16 12:25:14.112631 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:14.112534    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt" event={"ID":"47ad3acd-7f77-4703-b3be-73eb51c1c1fb","Type":"ContainerDied","Data":"92aee441df391bf3c789b010f499ae0b8f713aa379fb6bc459c436328e8fa9a5"}
Sep 16 12:25:14.112631 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:14.112556    2582 scope.go:117] "RemoveContainer" containerID="db59890412f7d61c03fffc000740e0241bcc4b0fe748fc357addf21a51e8fc7d"
Sep 16 12:25:14.122904 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:14.122880    2582 scope.go:117] "RemoveContainer" containerID="db59890412f7d61c03fffc000740e0241bcc4b0fe748fc357addf21a51e8fc7d"
Sep 16 12:25:14.123217 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:25:14.123197    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db59890412f7d61c03fffc000740e0241bcc4b0fe748fc357addf21a51e8fc7d\": container with ID starting with db59890412f7d61c03fffc000740e0241bcc4b0fe748fc357addf21a51e8fc7d not found: ID does not exist" containerID="db59890412f7d61c03fffc000740e0241bcc4b0fe748fc357addf21a51e8fc7d"
Sep 16 12:25:14.123284 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:14.123226    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db59890412f7d61c03fffc000740e0241bcc4b0fe748fc357addf21a51e8fc7d"} err="failed to get container status \"db59890412f7d61c03fffc000740e0241bcc4b0fe748fc357addf21a51e8fc7d\": rpc error: code = NotFound desc = could not find container \"db59890412f7d61c03fffc000740e0241bcc4b0fe748fc357addf21a51e8fc7d\": container with ID starting with db59890412f7d61c03fffc000740e0241bcc4b0fe748fc357addf21a51e8fc7d not found: ID does not exist"
Sep 16 12:25:14.135661 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:14.135635    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"]
Sep 16 12:25:14.139034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:14.139011    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f2bc0450-86755f56d-bf6pt"]
Sep 16 12:25:15.212438 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:15.212401    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47ad3acd-7f77-4703-b3be-73eb51c1c1fb" path="/var/lib/kubelet/pods/47ad3acd-7f77-4703-b3be-73eb51c1c1fb/volumes"
Sep 16 12:25:24.945247 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:24.945209    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"]
Sep 16 12:25:26.188697 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:26.188663    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"]
Sep 16 12:25:35.737499 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:35.737462    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"]
Sep 16 12:25:35.737939 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:35.737718    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk" podUID="adc4128a-7b81-45f1-8682-cb2351c9f9ee" containerName="maas-api" containerID="cri-o://09b1bb07e1640fe829c03003f5d85712a387dbb7c8561e68ceb1e866d905c25e" gracePeriod=30
Sep 16 12:25:35.985720 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:35.985696    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:25:36.109102 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.109059    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc4128a-7b81-45f1-8682-cb2351c9f9ee-openshift-service-ca\") pod \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\" (UID: \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\") "
Sep 16 12:25:36.109102 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.109107    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh9b9\" (UniqueName: \"kubernetes.io/projected/adc4128a-7b81-45f1-8682-cb2351c9f9ee-kube-api-access-jh9b9\") pod \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\" (UID: \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\") "
Sep 16 12:25:36.109339 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.109157    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adc4128a-7b81-45f1-8682-cb2351c9f9ee-metrics-certs\") pod \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\" (UID: \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\") "
Sep 16 12:25:36.109339 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.109200    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adc4128a-7b81-45f1-8682-cb2351c9f9ee-maas-api-tls\") pod \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\" (UID: \"adc4128a-7b81-45f1-8682-cb2351c9f9ee\") "
Sep 16 12:25:36.109548 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.109517    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc4128a-7b81-45f1-8682-cb2351c9f9ee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "adc4128a-7b81-45f1-8682-cb2351c9f9ee" (UID: "adc4128a-7b81-45f1-8682-cb2351c9f9ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:25:36.111615 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.111584    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc4128a-7b81-45f1-8682-cb2351c9f9ee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "adc4128a-7b81-45f1-8682-cb2351c9f9ee" (UID: "adc4128a-7b81-45f1-8682-cb2351c9f9ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:25:36.111729 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.111614    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc4128a-7b81-45f1-8682-cb2351c9f9ee-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "adc4128a-7b81-45f1-8682-cb2351c9f9ee" (UID: "adc4128a-7b81-45f1-8682-cb2351c9f9ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:25:36.111729 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.111640    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc4128a-7b81-45f1-8682-cb2351c9f9ee-kube-api-access-jh9b9" (OuterVolumeSpecName: "kube-api-access-jh9b9") pod "adc4128a-7b81-45f1-8682-cb2351c9f9ee" (UID: "adc4128a-7b81-45f1-8682-cb2351c9f9ee"). InnerVolumeSpecName "kube-api-access-jh9b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:25:36.210269 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.210224    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc4128a-7b81-45f1-8682-cb2351c9f9ee-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:36.210269 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.210259    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh9b9\" (UniqueName: \"kubernetes.io/projected/adc4128a-7b81-45f1-8682-cb2351c9f9ee-kube-api-access-jh9b9\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:36.210269 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.210269    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adc4128a-7b81-45f1-8682-cb2351c9f9ee-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:36.210269 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.210277    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adc4128a-7b81-45f1-8682-cb2351c9f9ee-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:36.243868 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.243833    2582 generic.go:356] "Generic (PLEG): container finished" podID="adc4128a-7b81-45f1-8682-cb2351c9f9ee" containerID="09b1bb07e1640fe829c03003f5d85712a387dbb7c8561e68ceb1e866d905c25e" exitCode=0
Sep 16 12:25:36.244031 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.243899    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"
Sep 16 12:25:36.244031 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.243913    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk" event={"ID":"adc4128a-7b81-45f1-8682-cb2351c9f9ee","Type":"ContainerDied","Data":"09b1bb07e1640fe829c03003f5d85712a387dbb7c8561e68ceb1e866d905c25e"}
Sep 16 12:25:36.244031 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.243967    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk" event={"ID":"adc4128a-7b81-45f1-8682-cb2351c9f9ee","Type":"ContainerDied","Data":"335ac9267ac34ec0fa5589afba1fa4c4492744ca0d5efb2b5347b55f21ea8885"}
Sep 16 12:25:36.244031 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.243991    2582 scope.go:117] "RemoveContainer" containerID="09b1bb07e1640fe829c03003f5d85712a387dbb7c8561e68ceb1e866d905c25e"
Sep 16 12:25:36.253968 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.253949    2582 scope.go:117] "RemoveContainer" containerID="09b1bb07e1640fe829c03003f5d85712a387dbb7c8561e68ceb1e866d905c25e"
Sep 16 12:25:36.254248 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:25:36.254224    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09b1bb07e1640fe829c03003f5d85712a387dbb7c8561e68ceb1e866d905c25e\": container with ID starting with 09b1bb07e1640fe829c03003f5d85712a387dbb7c8561e68ceb1e866d905c25e not found: ID does not exist" containerID="09b1bb07e1640fe829c03003f5d85712a387dbb7c8561e68ceb1e866d905c25e"
Sep 16 12:25:36.254325 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.254261    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b1bb07e1640fe829c03003f5d85712a387dbb7c8561e68ceb1e866d905c25e"} err="failed to get container status \"09b1bb07e1640fe829c03003f5d85712a387dbb7c8561e68ceb1e866d905c25e\": rpc error: code = NotFound desc = could not find container \"09b1bb07e1640fe829c03003f5d85712a387dbb7c8561e68ceb1e866d905c25e\": container with ID starting with 09b1bb07e1640fe829c03003f5d85712a387dbb7c8561e68ceb1e866d905c25e not found: ID does not exist"
Sep 16 12:25:36.265702 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.265677    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"]
Sep 16 12:25:36.271179 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:36.271156    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4915f8-db5b46b58-vlfmk"]
Sep 16 12:25:37.213433 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:37.213391    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adc4128a-7b81-45f1-8682-cb2351c9f9ee" path="/var/lib/kubelet/pods/adc4128a-7b81-45f1-8682-cb2351c9f9ee/volumes"
Sep 16 12:25:39.593355 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.593312    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"]
Sep 16 12:25:39.593911 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.593703    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ad3acd-7f77-4703-b3be-73eb51c1c1fb" containerName="maas-api"
Sep 16 12:25:39.593911 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.593715    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ad3acd-7f77-4703-b3be-73eb51c1c1fb" containerName="maas-api"
Sep 16 12:25:39.593911 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.593759    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adc4128a-7b81-45f1-8682-cb2351c9f9ee" containerName="maas-api"
Sep 16 12:25:39.593911 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.593768    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="adc4128a-7b81-45f1-8682-cb2351c9f9ee" containerName="maas-api"
Sep 16 12:25:39.593911 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.593833    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="adc4128a-7b81-45f1-8682-cb2351c9f9ee" containerName="maas-api"
Sep 16 12:25:39.593911 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.593847    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="47ad3acd-7f77-4703-b3be-73eb51c1c1fb" containerName="maas-api"
Sep 16 12:25:39.604973 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.599754    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:25:39.604973 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.603406    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-65d8af-serving-cert\""
Sep 16 12:25:39.610694 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.610399    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"]
Sep 16 12:25:39.747274 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.747236    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06773e03-fd60-43ae-aa35-8dd3b5f6a665-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl\" (UID: \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:25:39.747452 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.747292    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06773e03-fd60-43ae-aa35-8dd3b5f6a665-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl\" (UID: \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:25:39.747452 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.747358    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06773e03-fd60-43ae-aa35-8dd3b5f6a665-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl\" (UID: \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:25:39.747531 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.747473    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw46f\" (UniqueName: \"kubernetes.io/projected/06773e03-fd60-43ae-aa35-8dd3b5f6a665-kube-api-access-lw46f\") pod \"maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl\" (UID: \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:25:39.848791 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.848668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06773e03-fd60-43ae-aa35-8dd3b5f6a665-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl\" (UID: \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:25:39.848791 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.848711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06773e03-fd60-43ae-aa35-8dd3b5f6a665-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl\" (UID: \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:25:39.849050 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.848810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw46f\" (UniqueName: \"kubernetes.io/projected/06773e03-fd60-43ae-aa35-8dd3b5f6a665-kube-api-access-lw46f\") pod \"maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl\" (UID: \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:25:39.849050 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.848869    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06773e03-fd60-43ae-aa35-8dd3b5f6a665-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl\" (UID: \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:25:39.849594 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.849516    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06773e03-fd60-43ae-aa35-8dd3b5f6a665-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl\" (UID: \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:25:39.851356 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.851329    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06773e03-fd60-43ae-aa35-8dd3b5f6a665-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl\" (UID: \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:25:39.851498 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.851476    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06773e03-fd60-43ae-aa35-8dd3b5f6a665-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl\" (UID: \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:25:39.857224 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.857199    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw46f\" (UniqueName: \"kubernetes.io/projected/06773e03-fd60-43ae-aa35-8dd3b5f6a665-kube-api-access-lw46f\") pod \"maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl\" (UID: \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:25:39.916884 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.916847    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:25:39.965919 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:39.965846    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4" podUID="05d4c2e8-cf18-4221-b418-5b57bfa519a3" containerName="main" containerID="cri-o://b6a5a3ed1aa0a5c0094b356351abd5f770938625037ec16a07c6c011945ecd74" gracePeriod=45
Sep 16 12:25:40.204434 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.204395    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"]
Sep 16 12:25:40.205727 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:25:40.205695    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06773e03_fd60_43ae_aa35_8dd3b5f6a665.slice/crio-0825b081360cd6965925e2c4d99346799975bc52a136004af2188fd9d39b5f48 WatchSource:0}: Error finding container 0825b081360cd6965925e2c4d99346799975bc52a136004af2188fd9d39b5f48: Status 404 returned error can't find the container with id 0825b081360cd6965925e2c4d99346799975bc52a136004af2188fd9d39b5f48
Sep 16 12:25:40.214637 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.214609    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:40.267667 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.267633    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl" event={"ID":"06773e03-fd60-43ae-aa35-8dd3b5f6a665","Type":"ContainerStarted","Data":"0825b081360cd6965925e2c4d99346799975bc52a136004af2188fd9d39b5f48"}
Sep 16 12:25:40.269116 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.269092    2582 generic.go:356] "Generic (PLEG): container finished" podID="05d4c2e8-cf18-4221-b418-5b57bfa519a3" containerID="b6a5a3ed1aa0a5c0094b356351abd5f770938625037ec16a07c6c011945ecd74" exitCode=0
Sep 16 12:25:40.269243 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.269133    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"
Sep 16 12:25:40.269243 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.269134    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4" event={"ID":"05d4c2e8-cf18-4221-b418-5b57bfa519a3","Type":"ContainerDied","Data":"b6a5a3ed1aa0a5c0094b356351abd5f770938625037ec16a07c6c011945ecd74"}
Sep 16 12:25:40.269333 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.269267    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4" event={"ID":"05d4c2e8-cf18-4221-b418-5b57bfa519a3","Type":"ContainerDied","Data":"d457458ddc3d6fc6ff192396bbbadf42d5a0bf3742b58ea701dd2fa4a63f6a2f"}
Sep 16 12:25:40.269333 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.269293    2582 scope.go:117] "RemoveContainer" containerID="b6a5a3ed1aa0a5c0094b356351abd5f770938625037ec16a07c6c011945ecd74"
Sep 16 12:25:40.278525 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.278469    2582 scope.go:117] "RemoveContainer" containerID="b6a5a3ed1aa0a5c0094b356351abd5f770938625037ec16a07c6c011945ecd74"
Sep 16 12:25:40.278785 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:25:40.278767    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a5a3ed1aa0a5c0094b356351abd5f770938625037ec16a07c6c011945ecd74\": container with ID starting with b6a5a3ed1aa0a5c0094b356351abd5f770938625037ec16a07c6c011945ecd74 not found: ID does not exist" containerID="b6a5a3ed1aa0a5c0094b356351abd5f770938625037ec16a07c6c011945ecd74"
Sep 16 12:25:40.278869 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.278794    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a5a3ed1aa0a5c0094b356351abd5f770938625037ec16a07c6c011945ecd74"} err="failed to get container status \"b6a5a3ed1aa0a5c0094b356351abd5f770938625037ec16a07c6c011945ecd74\": rpc error: code = NotFound desc = could not find container \"b6a5a3ed1aa0a5c0094b356351abd5f770938625037ec16a07c6c011945ecd74\": container with ID starting with b6a5a3ed1aa0a5c0094b356351abd5f770938625037ec16a07c6c011945ecd74 not found: ID does not exist"
Sep 16 12:25:40.352782 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.352746    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-model-cache\") pod \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") "
Sep 16 12:25:40.352982 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.352818    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f46s5\" (UniqueName: \"kubernetes.io/projected/05d4c2e8-cf18-4221-b418-5b57bfa519a3-kube-api-access-f46s5\") pod \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") "
Sep 16 12:25:40.352982 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.352863    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05d4c2e8-cf18-4221-b418-5b57bfa519a3-tls-certs\") pod \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") "
Sep 16 12:25:40.352982 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.352889    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-dshm\") pod \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") "
Sep 16 12:25:40.352982 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.352959    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-tmp-dir\") pod \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") "
Sep 16 12:25:40.353214 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.353029    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-home\") pod \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\" (UID: \"05d4c2e8-cf18-4221-b418-5b57bfa519a3\") "
Sep 16 12:25:40.353214 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.353047    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "05d4c2e8-cf18-4221-b418-5b57bfa519a3" (UID: "05d4c2e8-cf18-4221-b418-5b57bfa519a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:40.353305 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.353250    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05d4c2e8-cf18-4221-b418-5b57bfa519a3" (UID: "05d4c2e8-cf18-4221-b418-5b57bfa519a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:40.353392 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.353331    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-home" (OuterVolumeSpecName: "home") pod "05d4c2e8-cf18-4221-b418-5b57bfa519a3" (UID: "05d4c2e8-cf18-4221-b418-5b57bfa519a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:40.353504 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.353484    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:40.353569 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.353510    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:40.353569 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.353524    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:40.355626 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.355602    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-dshm" (OuterVolumeSpecName: "dshm") pod "05d4c2e8-cf18-4221-b418-5b57bfa519a3" (UID: "05d4c2e8-cf18-4221-b418-5b57bfa519a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:40.355781 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.355631    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d4c2e8-cf18-4221-b418-5b57bfa519a3-kube-api-access-f46s5" (OuterVolumeSpecName: "kube-api-access-f46s5") pod "05d4c2e8-cf18-4221-b418-5b57bfa519a3" (UID: "05d4c2e8-cf18-4221-b418-5b57bfa519a3"). InnerVolumeSpecName "kube-api-access-f46s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:25:40.355858 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.355823    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d4c2e8-cf18-4221-b418-5b57bfa519a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05d4c2e8-cf18-4221-b418-5b57bfa519a3" (UID: "05d4c2e8-cf18-4221-b418-5b57bfa519a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:25:40.454306 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.454213    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05d4c2e8-cf18-4221-b418-5b57bfa519a3-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:40.454306 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.454246    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f46s5\" (UniqueName: \"kubernetes.io/projected/05d4c2e8-cf18-4221-b418-5b57bfa519a3-kube-api-access-f46s5\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:40.454306 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.454258    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05d4c2e8-cf18-4221-b418-5b57bfa519a3-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:40.591195 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.591160    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"]
Sep 16 12:25:40.595036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:40.595007    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-4915f8/explicit-model-4915f8-kserve-7fd978b59f-wt2d4"]
Sep 16 12:25:41.209165 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.209044    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj" podUID="42829500-7641-45d5-956d-60e26554756e" containerName="main" containerID="cri-o://a345c56b53281db61cd4b73cfa3421694302ea5995c4620a54794cf5ef232640" gracePeriod=45
Sep 16 12:25:41.214006 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.213861    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05d4c2e8-cf18-4221-b418-5b57bfa519a3" path="/var/lib/kubelet/pods/05d4c2e8-cf18-4221-b418-5b57bfa519a3/volumes"
Sep 16 12:25:41.276521 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.276488    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl" event={"ID":"06773e03-fd60-43ae-aa35-8dd3b5f6a665","Type":"ContainerStarted","Data":"5970a6a4e4269604862882c969a961140535b9a6fa3dfe16b0fe067a958e8589"}
Sep 16 12:25:41.276705 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.276589    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:25:41.301141 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.301080    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl" podStartSLOduration=1.935920811 podStartE2EDuration="2.30106374s" totalImagesPullingTime="365.142929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:25:41.298926523 +0000 UTC m=+1500.549464073" watchObservedRunningTime="2026-09-16 12:25:41.30106374 +0000 UTC m=+1500.551601289"
Sep 16 12:25:41.467719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.467650    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:25:41.564823 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.564786    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stfks\" (UniqueName: \"kubernetes.io/projected/42829500-7641-45d5-956d-60e26554756e-kube-api-access-stfks\") pod \"42829500-7641-45d5-956d-60e26554756e\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") "
Sep 16 12:25:41.565045 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.564896    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-model-cache\") pod \"42829500-7641-45d5-956d-60e26554756e\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") "
Sep 16 12:25:41.565045 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.564938    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42829500-7641-45d5-956d-60e26554756e-tls-certs\") pod \"42829500-7641-45d5-956d-60e26554756e\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") "
Sep 16 12:25:41.565045 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.564970    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-home\") pod \"42829500-7641-45d5-956d-60e26554756e\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") "
Sep 16 12:25:41.565045 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.565034    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-dshm\") pod \"42829500-7641-45d5-956d-60e26554756e\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") "
Sep 16 12:25:41.565265 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.565060    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-tmp-dir\") pod \"42829500-7641-45d5-956d-60e26554756e\" (UID: \"42829500-7641-45d5-956d-60e26554756e\") "
Sep 16 12:25:41.565265 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.565140    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-model-cache" (OuterVolumeSpecName: "model-cache") pod "42829500-7641-45d5-956d-60e26554756e" (UID: "42829500-7641-45d5-956d-60e26554756e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:41.565368 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.565320    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-home" (OuterVolumeSpecName: "home") pod "42829500-7641-45d5-956d-60e26554756e" (UID: "42829500-7641-45d5-956d-60e26554756e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:41.565368 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.565348    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42829500-7641-45d5-956d-60e26554756e" (UID: "42829500-7641-45d5-956d-60e26554756e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:41.565518 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.565502    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:41.565576 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.565522    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:41.565576 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.565537    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:41.567418 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.567396    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-dshm" (OuterVolumeSpecName: "dshm") pod "42829500-7641-45d5-956d-60e26554756e" (UID: "42829500-7641-45d5-956d-60e26554756e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:41.567687 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.567665    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42829500-7641-45d5-956d-60e26554756e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42829500-7641-45d5-956d-60e26554756e" (UID: "42829500-7641-45d5-956d-60e26554756e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:25:41.567687 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.567672    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42829500-7641-45d5-956d-60e26554756e-kube-api-access-stfks" (OuterVolumeSpecName: "kube-api-access-stfks") pod "42829500-7641-45d5-956d-60e26554756e" (UID: "42829500-7641-45d5-956d-60e26554756e"). InnerVolumeSpecName "kube-api-access-stfks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:25:41.666707 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.666657    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42829500-7641-45d5-956d-60e26554756e-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:41.666707 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.666698    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42829500-7641-45d5-956d-60e26554756e-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:41.666707 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:41.666709    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stfks\" (UniqueName: \"kubernetes.io/projected/42829500-7641-45d5-956d-60e26554756e-kube-api-access-stfks\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:42.285027 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:42.284989    2582 generic.go:356] "Generic (PLEG): container finished" podID="42829500-7641-45d5-956d-60e26554756e" containerID="a345c56b53281db61cd4b73cfa3421694302ea5995c4620a54794cf5ef232640" exitCode=0
Sep 16 12:25:42.285243 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:42.285060    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"
Sep 16 12:25:42.285243 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:42.285074    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj" event={"ID":"42829500-7641-45d5-956d-60e26554756e","Type":"ContainerDied","Data":"a345c56b53281db61cd4b73cfa3421694302ea5995c4620a54794cf5ef232640"}
Sep 16 12:25:42.285243 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:42.285119    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj" event={"ID":"42829500-7641-45d5-956d-60e26554756e","Type":"ContainerDied","Data":"fdf0e44a4140ef9c684a4532b2292087b0ad0645d4c39fef4ac98a9674931f75"}
Sep 16 12:25:42.285243 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:42.285138    2582 scope.go:117] "RemoveContainer" containerID="a345c56b53281db61cd4b73cfa3421694302ea5995c4620a54794cf5ef232640"
Sep 16 12:25:42.294924 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:42.294902    2582 scope.go:117] "RemoveContainer" containerID="a345c56b53281db61cd4b73cfa3421694302ea5995c4620a54794cf5ef232640"
Sep 16 12:25:42.295232 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:25:42.295208    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a345c56b53281db61cd4b73cfa3421694302ea5995c4620a54794cf5ef232640\": container with ID starting with a345c56b53281db61cd4b73cfa3421694302ea5995c4620a54794cf5ef232640 not found: ID does not exist" containerID="a345c56b53281db61cd4b73cfa3421694302ea5995c4620a54794cf5ef232640"
Sep 16 12:25:42.295288 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:42.295240    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a345c56b53281db61cd4b73cfa3421694302ea5995c4620a54794cf5ef232640"} err="failed to get container status \"a345c56b53281db61cd4b73cfa3421694302ea5995c4620a54794cf5ef232640\": rpc error: code = NotFound desc = could not find container \"a345c56b53281db61cd4b73cfa3421694302ea5995c4620a54794cf5ef232640\": container with ID starting with a345c56b53281db61cd4b73cfa3421694302ea5995c4620a54794cf5ef232640 not found: ID does not exist"
Sep 16 12:25:42.307259 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:42.307225    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"]
Sep 16 12:25:42.314540 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:42.314515    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9a49af/test-model-9a49af-kserve-66d69598b9-6vrlj"]
Sep 16 12:25:43.213364 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:43.213329    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42829500-7641-45d5-956d-60e26554756e" path="/var/lib/kubelet/pods/42829500-7641-45d5-956d-60e26554756e/volumes"
Sep 16 12:25:46.487080 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:46.487047    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"]
Sep 16 12:25:47.291850 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:47.291804    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:25:57.528436 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:57.528399    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"]
Sep 16 12:25:57.530977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:57.528613    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s" podUID="33e408e5-9ee5-4fcb-ba46-64f031e812f2" containerName="maas-api" containerID="cri-o://8a397118c4e5bdef9ae02f34cbaff1f9ef18486ad4d8a479c95e78466897515c" gracePeriod=30
Sep 16 12:25:57.784320 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:57.784255    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:25:57.820880 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:57.820841    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e408e5-9ee5-4fcb-ba46-64f031e812f2-maas-api-tls\") pod \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\" (UID: \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\") "
Sep 16 12:25:57.821092 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:57.820960    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e408e5-9ee5-4fcb-ba46-64f031e812f2-openshift-service-ca\") pod \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\" (UID: \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\") "
Sep 16 12:25:57.821092 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:57.821027    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e408e5-9ee5-4fcb-ba46-64f031e812f2-metrics-certs\") pod \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\" (UID: \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\") "
Sep 16 12:25:57.821092 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:57.821056    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlxbk\" (UniqueName: \"kubernetes.io/projected/33e408e5-9ee5-4fcb-ba46-64f031e812f2-kube-api-access-tlxbk\") pod \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\" (UID: \"33e408e5-9ee5-4fcb-ba46-64f031e812f2\") "
Sep 16 12:25:57.821613 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:57.821524    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e408e5-9ee5-4fcb-ba46-64f031e812f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33e408e5-9ee5-4fcb-ba46-64f031e812f2" (UID: "33e408e5-9ee5-4fcb-ba46-64f031e812f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:25:57.824315 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:57.824268    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e408e5-9ee5-4fcb-ba46-64f031e812f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33e408e5-9ee5-4fcb-ba46-64f031e812f2" (UID: "33e408e5-9ee5-4fcb-ba46-64f031e812f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:25:57.824573 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:57.824541    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e408e5-9ee5-4fcb-ba46-64f031e812f2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "33e408e5-9ee5-4fcb-ba46-64f031e812f2" (UID: "33e408e5-9ee5-4fcb-ba46-64f031e812f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:25:57.824932 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:57.824900    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e408e5-9ee5-4fcb-ba46-64f031e812f2-kube-api-access-tlxbk" (OuterVolumeSpecName: "kube-api-access-tlxbk") pod "33e408e5-9ee5-4fcb-ba46-64f031e812f2" (UID: "33e408e5-9ee5-4fcb-ba46-64f031e812f2"). InnerVolumeSpecName "kube-api-access-tlxbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:25:57.922606 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:57.922571    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e408e5-9ee5-4fcb-ba46-64f031e812f2-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:57.922606 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:57.922602    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e408e5-9ee5-4fcb-ba46-64f031e812f2-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:57.922606 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:57.922611    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e408e5-9ee5-4fcb-ba46-64f031e812f2-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:57.922864 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:57.922620    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlxbk\" (UniqueName: \"kubernetes.io/projected/33e408e5-9ee5-4fcb-ba46-64f031e812f2-kube-api-access-tlxbk\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:25:58.378618 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:58.378582    2582 generic.go:356] "Generic (PLEG): container finished" podID="33e408e5-9ee5-4fcb-ba46-64f031e812f2" containerID="8a397118c4e5bdef9ae02f34cbaff1f9ef18486ad4d8a479c95e78466897515c" exitCode=0
Sep 16 12:25:58.378831 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:58.378639    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"
Sep 16 12:25:58.378831 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:58.378674    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s" event={"ID":"33e408e5-9ee5-4fcb-ba46-64f031e812f2","Type":"ContainerDied","Data":"8a397118c4e5bdef9ae02f34cbaff1f9ef18486ad4d8a479c95e78466897515c"}
Sep 16 12:25:58.378831 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:58.378722    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s" event={"ID":"33e408e5-9ee5-4fcb-ba46-64f031e812f2","Type":"ContainerDied","Data":"f9ba8f9348523e61c9ee1e685a7dbf1cff4260f5115be3f642c17fb485167ec7"}
Sep 16 12:25:58.378831 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:58.378772    2582 scope.go:117] "RemoveContainer" containerID="8a397118c4e5bdef9ae02f34cbaff1f9ef18486ad4d8a479c95e78466897515c"
Sep 16 12:25:58.388963 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:58.388890    2582 scope.go:117] "RemoveContainer" containerID="8a397118c4e5bdef9ae02f34cbaff1f9ef18486ad4d8a479c95e78466897515c"
Sep 16 12:25:58.389329 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:25:58.389306    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a397118c4e5bdef9ae02f34cbaff1f9ef18486ad4d8a479c95e78466897515c\": container with ID starting with 8a397118c4e5bdef9ae02f34cbaff1f9ef18486ad4d8a479c95e78466897515c not found: ID does not exist" containerID="8a397118c4e5bdef9ae02f34cbaff1f9ef18486ad4d8a479c95e78466897515c"
Sep 16 12:25:58.389427 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:58.389342    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a397118c4e5bdef9ae02f34cbaff1f9ef18486ad4d8a479c95e78466897515c"} err="failed to get container status \"8a397118c4e5bdef9ae02f34cbaff1f9ef18486ad4d8a479c95e78466897515c\": rpc error: code = NotFound desc = could not find container \"8a397118c4e5bdef9ae02f34cbaff1f9ef18486ad4d8a479c95e78466897515c\": container with ID starting with 8a397118c4e5bdef9ae02f34cbaff1f9ef18486ad4d8a479c95e78466897515c not found: ID does not exist"
Sep 16 12:25:58.403428 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:58.403394    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"]
Sep 16 12:25:58.407616 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:58.407592    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9a49af-675b4b5d5b-z856s"]
Sep 16 12:25:59.218078 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.218046    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33e408e5-9ee5-4fcb-ba46-64f031e812f2" path="/var/lib/kubelet/pods/33e408e5-9ee5-4fcb-ba46-64f031e812f2/volumes"
Sep 16 12:25:59.814926 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.814889    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"]
Sep 16 12:25:59.815415 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.815394    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05d4c2e8-cf18-4221-b418-5b57bfa519a3" containerName="main"
Sep 16 12:25:59.815415 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.815415    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="05d4c2e8-cf18-4221-b418-5b57bfa519a3" containerName="main"
Sep 16 12:25:59.815563 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.815434    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e408e5-9ee5-4fcb-ba46-64f031e812f2" containerName="maas-api"
Sep 16 12:25:59.815563 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.815443    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e408e5-9ee5-4fcb-ba46-64f031e812f2" containerName="maas-api"
Sep 16 12:25:59.815563 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.815460    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42829500-7641-45d5-956d-60e26554756e" containerName="main"
Sep 16 12:25:59.815563 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.815466    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="42829500-7641-45d5-956d-60e26554756e" containerName="main"
Sep 16 12:25:59.815563 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.815531    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="42829500-7641-45d5-956d-60e26554756e" containerName="main"
Sep 16 12:25:59.815563 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.815540    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="33e408e5-9ee5-4fcb-ba46-64f031e812f2" containerName="maas-api"
Sep 16 12:25:59.815563 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.815548    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="05d4c2e8-cf18-4221-b418-5b57bfa519a3" containerName="main"
Sep 16 12:25:59.820051 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.820033    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:25:59.822868 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.822837    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-f9133e-serving-cert\""
Sep 16 12:25:59.831050 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.831026    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"]
Sep 16 12:25:59.940559 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.940519    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft\" (UID: \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:25:59.940559 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.940573    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4m62\" (UniqueName: \"kubernetes.io/projected/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-kube-api-access-k4m62\") pod \"maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft\" (UID: \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:25:59.940811 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.940654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft\" (UID: \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:25:59.940811 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:25:59.940712    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft\" (UID: \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:26:00.041355 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:00.041309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft\" (UID: \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:26:00.041532 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:00.041383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft\" (UID: \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:26:00.041532 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:00.041417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft\" (UID: \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:26:00.041532 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:00.041451    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4m62\" (UniqueName: \"kubernetes.io/projected/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-kube-api-access-k4m62\") pod \"maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft\" (UID: \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:26:00.041532 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:00.041521    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-serving-cert: secret "maas-api-e2e-shared-b-w2-f9133e-serving-cert" not found
Sep 16 12:26:00.041677 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:00.041606    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-maas-api-tls podName:cc6c4a13-2d36-4c6b-a3e7-0d67996e3103 nodeName:}" failed. No retries permitted until 2026-09-16 12:26:00.541584923 +0000 UTC m=+1519.792122468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-maas-api-tls") pod "maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft" (UID: "cc6c4a13-2d36-4c6b-a3e7-0d67996e3103") : secret "maas-api-e2e-shared-b-w2-f9133e-serving-cert" not found
Sep 16 12:26:00.042112 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:00.042095    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft\" (UID: \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:26:00.044022 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:00.043993    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft\" (UID: \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:26:00.049864 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:00.049848    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4m62\" (UniqueName: \"kubernetes.io/projected/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-kube-api-access-k4m62\") pod \"maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft\" (UID: \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:26:00.546796 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:00.546759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft\" (UID: \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:26:00.549318 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:00.549287    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft\" (UID: \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:26:00.731454 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:00.731413    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:26:01.008819 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.008789    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"]
Sep 16 12:26:01.010469 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:26:01.010440    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6c4a13_2d36_4c6b_a3e7_0d67996e3103.slice/crio-0353c8b0a7b95ba270ac21ca4c5598f2886c361f44c9637196ffdd841880f554 WatchSource:0}: Error finding container 0353c8b0a7b95ba270ac21ca4c5598f2886c361f44c9637196ffdd841880f554: Status 404 returned error can't find the container with id 0353c8b0a7b95ba270ac21ca4c5598f2886c361f44c9637196ffdd841880f554
Sep 16 12:26:01.397929 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.397897    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft" event={"ID":"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103","Type":"ContainerStarted","Data":"0353c8b0a7b95ba270ac21ca4c5598f2886c361f44c9637196ffdd841880f554"}
Sep 16 12:26:01.508183 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.508117    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5" podUID="a5a9dfed-3f79-4f97-83a1-93f376fd302f" containerName="main" containerID="cri-o://f3a14b7d6b113b2988d8996dedded5197026c05b73f9c7d59222b1348e9dd58b" gracePeriod=45
Sep 16 12:26:01.756118 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.756093    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:26:01.860330 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.860298    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-tmp-dir\") pod \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") "
Sep 16 12:26:01.860515 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.860349    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-model-cache\") pod \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") "
Sep 16 12:26:01.860515 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.860449    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-home\") pod \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") "
Sep 16 12:26:01.860515 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.860502    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-dshm\") pod \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") "
Sep 16 12:26:01.860696 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.860554    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk2xw\" (UniqueName: \"kubernetes.io/projected/a5a9dfed-3f79-4f97-83a1-93f376fd302f-kube-api-access-rk2xw\") pod \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") "
Sep 16 12:26:01.860696 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.860584    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a9dfed-3f79-4f97-83a1-93f376fd302f-tls-certs\") pod \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\" (UID: \"a5a9dfed-3f79-4f97-83a1-93f376fd302f\") "
Sep 16 12:26:01.860696 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.860617    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5a9dfed-3f79-4f97-83a1-93f376fd302f" (UID: "a5a9dfed-3f79-4f97-83a1-93f376fd302f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:26:01.860696 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.860627    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5a9dfed-3f79-4f97-83a1-93f376fd302f" (UID: "a5a9dfed-3f79-4f97-83a1-93f376fd302f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:26:01.860935 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.860695    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-home" (OuterVolumeSpecName: "home") pod "a5a9dfed-3f79-4f97-83a1-93f376fd302f" (UID: "a5a9dfed-3f79-4f97-83a1-93f376fd302f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:26:01.860990 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.860940    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:26:01.860990 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.860961    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:26:01.860990 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.860976    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:26:01.862991 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.862969    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-dshm" (OuterVolumeSpecName: "dshm") pod "a5a9dfed-3f79-4f97-83a1-93f376fd302f" (UID: "a5a9dfed-3f79-4f97-83a1-93f376fd302f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:26:01.863077 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.863027    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a9dfed-3f79-4f97-83a1-93f376fd302f-kube-api-access-rk2xw" (OuterVolumeSpecName: "kube-api-access-rk2xw") pod "a5a9dfed-3f79-4f97-83a1-93f376fd302f" (UID: "a5a9dfed-3f79-4f97-83a1-93f376fd302f"). InnerVolumeSpecName "kube-api-access-rk2xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:26:01.863119 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.863088    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a9dfed-3f79-4f97-83a1-93f376fd302f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5a9dfed-3f79-4f97-83a1-93f376fd302f" (UID: "a5a9dfed-3f79-4f97-83a1-93f376fd302f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:26:01.961550 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.961463    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a9dfed-3f79-4f97-83a1-93f376fd302f-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:26:01.961550 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.961494    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk2xw\" (UniqueName: \"kubernetes.io/projected/a5a9dfed-3f79-4f97-83a1-93f376fd302f-kube-api-access-rk2xw\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:26:01.961550 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:01.961504    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a9dfed-3f79-4f97-83a1-93f376fd302f-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:26:02.405792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:02.405716    2582 generic.go:356] "Generic (PLEG): container finished" podID="a5a9dfed-3f79-4f97-83a1-93f376fd302f" containerID="f3a14b7d6b113b2988d8996dedded5197026c05b73f9c7d59222b1348e9dd58b" exitCode=0
Sep 16 12:26:02.405792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:02.405791    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"
Sep 16 12:26:02.406033 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:02.405812    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5" event={"ID":"a5a9dfed-3f79-4f97-83a1-93f376fd302f","Type":"ContainerDied","Data":"f3a14b7d6b113b2988d8996dedded5197026c05b73f9c7d59222b1348e9dd58b"}
Sep 16 12:26:02.406033 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:02.405863    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5" event={"ID":"a5a9dfed-3f79-4f97-83a1-93f376fd302f","Type":"ContainerDied","Data":"d4a87b882dbcbfd01776b7ae907d83d589eeb6c185db0c765a12cf48aa8e6973"}
Sep 16 12:26:02.406033 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:02.405880    2582 scope.go:117] "RemoveContainer" containerID="f3a14b7d6b113b2988d8996dedded5197026c05b73f9c7d59222b1348e9dd58b"
Sep 16 12:26:02.407833 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:02.407773    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft" event={"ID":"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103","Type":"ContainerStarted","Data":"af222f7f3eae91cb845ac7d03008d19508ada195008e825b3a4aae2159bbca95"}
Sep 16 12:26:02.407957 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:02.407864    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:26:02.417107 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:02.417086    2582 scope.go:117] "RemoveContainer" containerID="f3a14b7d6b113b2988d8996dedded5197026c05b73f9c7d59222b1348e9dd58b"
Sep 16 12:26:02.417389 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:02.417371    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a14b7d6b113b2988d8996dedded5197026c05b73f9c7d59222b1348e9dd58b\": container with ID starting with f3a14b7d6b113b2988d8996dedded5197026c05b73f9c7d59222b1348e9dd58b not found: ID does not exist" containerID="f3a14b7d6b113b2988d8996dedded5197026c05b73f9c7d59222b1348e9dd58b"
Sep 16 12:26:02.417439 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:02.417401    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a14b7d6b113b2988d8996dedded5197026c05b73f9c7d59222b1348e9dd58b"} err="failed to get container status \"f3a14b7d6b113b2988d8996dedded5197026c05b73f9c7d59222b1348e9dd58b\": rpc error: code = NotFound desc = could not find container \"f3a14b7d6b113b2988d8996dedded5197026c05b73f9c7d59222b1348e9dd58b\": container with ID starting with f3a14b7d6b113b2988d8996dedded5197026c05b73f9c7d59222b1348e9dd58b not found: ID does not exist"
Sep 16 12:26:02.430137 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:02.430094    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft" podStartSLOduration=3.094593781 podStartE2EDuration="3.430079288s" totalImagesPullingTime="335.485507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:02.427991165 +0000 UTC m=+1521.678528716" watchObservedRunningTime="2026-09-16 12:26:02.430079288 +0000 UTC m=+1521.680616866"
Sep 16 12:26:02.447169 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:02.447145    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"]
Sep 16 12:26:02.450915 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:02.450894    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4bef4d/test-model-4bef4d-kserve-7f54468f55-zs7k5"]
Sep 16 12:26:03.213997 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:03.213966    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5a9dfed-3f79-4f97-83a1-93f376fd302f" path="/var/lib/kubelet/pods/a5a9dfed-3f79-4f97-83a1-93f376fd302f/volumes"
Sep 16 12:26:08.422050 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:08.422018    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:26:15.835391 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:15.835355    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"]
Sep 16 12:26:15.835885 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:15.835686    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7" podUID="3d62ba3d-5219-4153-b13a-ec54a22e919a" containerName="maas-api" containerID="cri-o://83b3beea9d1e836feec94f5559d3f9a7ca7b0f0ff46216cdfdc02b0567720a41" gracePeriod=30
Sep 16 12:26:16.086995 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.086936    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:26:16.201504 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.201469    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d62ba3d-5219-4153-b13a-ec54a22e919a-metrics-certs\") pod \"3d62ba3d-5219-4153-b13a-ec54a22e919a\" (UID: \"3d62ba3d-5219-4153-b13a-ec54a22e919a\") "
Sep 16 12:26:16.201707 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.201520    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d62ba3d-5219-4153-b13a-ec54a22e919a-openshift-service-ca\") pod \"3d62ba3d-5219-4153-b13a-ec54a22e919a\" (UID: \"3d62ba3d-5219-4153-b13a-ec54a22e919a\") "
Sep 16 12:26:16.201707 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.201601    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-648lp\" (UniqueName: \"kubernetes.io/projected/3d62ba3d-5219-4153-b13a-ec54a22e919a-kube-api-access-648lp\") pod \"3d62ba3d-5219-4153-b13a-ec54a22e919a\" (UID: \"3d62ba3d-5219-4153-b13a-ec54a22e919a\") "
Sep 16 12:26:16.201707 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.201617    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d62ba3d-5219-4153-b13a-ec54a22e919a-maas-api-tls\") pod \"3d62ba3d-5219-4153-b13a-ec54a22e919a\" (UID: \"3d62ba3d-5219-4153-b13a-ec54a22e919a\") "
Sep 16 12:26:16.202028 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.201995    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d62ba3d-5219-4153-b13a-ec54a22e919a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3d62ba3d-5219-4153-b13a-ec54a22e919a" (UID: "3d62ba3d-5219-4153-b13a-ec54a22e919a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:26:16.204002 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.203976    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d62ba3d-5219-4153-b13a-ec54a22e919a-kube-api-access-648lp" (OuterVolumeSpecName: "kube-api-access-648lp") pod "3d62ba3d-5219-4153-b13a-ec54a22e919a" (UID: "3d62ba3d-5219-4153-b13a-ec54a22e919a"). InnerVolumeSpecName "kube-api-access-648lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:26:16.204121 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.204037    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d62ba3d-5219-4153-b13a-ec54a22e919a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3d62ba3d-5219-4153-b13a-ec54a22e919a" (UID: "3d62ba3d-5219-4153-b13a-ec54a22e919a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:26:16.204121 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.204072    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d62ba3d-5219-4153-b13a-ec54a22e919a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3d62ba3d-5219-4153-b13a-ec54a22e919a" (UID: "3d62ba3d-5219-4153-b13a-ec54a22e919a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:26:16.303066 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.303029    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d62ba3d-5219-4153-b13a-ec54a22e919a-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:26:16.303066 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.303058    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d62ba3d-5219-4153-b13a-ec54a22e919a-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:26:16.303066 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.303068    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-648lp\" (UniqueName: \"kubernetes.io/projected/3d62ba3d-5219-4153-b13a-ec54a22e919a-kube-api-access-648lp\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:26:16.303309 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.303078    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d62ba3d-5219-4153-b13a-ec54a22e919a-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:26:16.496713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.496678    2582 generic.go:356] "Generic (PLEG): container finished" podID="3d62ba3d-5219-4153-b13a-ec54a22e919a" containerID="83b3beea9d1e836feec94f5559d3f9a7ca7b0f0ff46216cdfdc02b0567720a41" exitCode=0
Sep 16 12:26:16.496953 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.496767    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"
Sep 16 12:26:16.496953 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.496777    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7" event={"ID":"3d62ba3d-5219-4153-b13a-ec54a22e919a","Type":"ContainerDied","Data":"83b3beea9d1e836feec94f5559d3f9a7ca7b0f0ff46216cdfdc02b0567720a41"}
Sep 16 12:26:16.496953 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.496834    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7" event={"ID":"3d62ba3d-5219-4153-b13a-ec54a22e919a","Type":"ContainerDied","Data":"1f06db215169da985d30308388ac78e17a9da5fedd124dc72971df35222e12fe"}
Sep 16 12:26:16.496953 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.496858    2582 scope.go:117] "RemoveContainer" containerID="83b3beea9d1e836feec94f5559d3f9a7ca7b0f0ff46216cdfdc02b0567720a41"
Sep 16 12:26:16.506638 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.506617    2582 scope.go:117] "RemoveContainer" containerID="83b3beea9d1e836feec94f5559d3f9a7ca7b0f0ff46216cdfdc02b0567720a41"
Sep 16 12:26:16.506955 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:16.506936    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b3beea9d1e836feec94f5559d3f9a7ca7b0f0ff46216cdfdc02b0567720a41\": container with ID starting with 83b3beea9d1e836feec94f5559d3f9a7ca7b0f0ff46216cdfdc02b0567720a41 not found: ID does not exist" containerID="83b3beea9d1e836feec94f5559d3f9a7ca7b0f0ff46216cdfdc02b0567720a41"
Sep 16 12:26:16.507012 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.506969    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b3beea9d1e836feec94f5559d3f9a7ca7b0f0ff46216cdfdc02b0567720a41"} err="failed to get container status \"83b3beea9d1e836feec94f5559d3f9a7ca7b0f0ff46216cdfdc02b0567720a41\": rpc error: code = NotFound desc = could not find container \"83b3beea9d1e836feec94f5559d3f9a7ca7b0f0ff46216cdfdc02b0567720a41\": container with ID starting with 83b3beea9d1e836feec94f5559d3f9a7ca7b0f0ff46216cdfdc02b0567720a41 not found: ID does not exist"
Sep 16 12:26:16.519056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.519028    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"]
Sep 16 12:26:16.522845 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:16.522817    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4bef4d-85756b755b-49vv7"]
Sep 16 12:26:17.213078 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:17.213043    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d62ba3d-5219-4153-b13a-ec54a22e919a" path="/var/lib/kubelet/pods/3d62ba3d-5219-4153-b13a-ec54a22e919a/volumes"
Sep 16 12:26:22.476489 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:22.476449    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-4d114a"]
Sep 16 12:26:22.476930 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:22.476897    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d62ba3d-5219-4153-b13a-ec54a22e919a" containerName="maas-api"
Sep 16 12:26:22.476930 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:22.476912    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d62ba3d-5219-4153-b13a-ec54a22e919a" containerName="maas-api"
Sep 16 12:26:22.476930 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:22.476922    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5a9dfed-3f79-4f97-83a1-93f376fd302f" containerName="main"
Sep 16 12:26:22.476930 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:22.476927    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5a9dfed-3f79-4f97-83a1-93f376fd302f" containerName="main"
Sep 16 12:26:22.477064 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:22.476994    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5a9dfed-3f79-4f97-83a1-93f376fd302f" containerName="main"
Sep 16 12:26:22.477064 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:22.477002    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d62ba3d-5219-4153-b13a-ec54a22e919a" containerName="maas-api"
Sep 16 12:26:22.481591 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:22.481572    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-4d114a"
Sep 16 12:26:22.484919 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:22.484894    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h8vfr\""
Sep 16 12:26:22.487326 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:22.487303    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-4d114a"]
Sep 16 12:26:22.666081 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:22.666043    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mwt5\" (UniqueName: \"kubernetes.io/projected/ded73ea2-f332-466e-b0cc-68f10fe061ae-kube-api-access-8mwt5\") pod \"test-curl-7564-4d114a\" (UID: \"ded73ea2-f332-466e-b0cc-68f10fe061ae\") " pod="opendatahub/test-curl-7564-4d114a"
Sep 16 12:26:22.767073 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:22.766983    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mwt5\" (UniqueName: \"kubernetes.io/projected/ded73ea2-f332-466e-b0cc-68f10fe061ae-kube-api-access-8mwt5\") pod \"test-curl-7564-4d114a\" (UID: \"ded73ea2-f332-466e-b0cc-68f10fe061ae\") " pod="opendatahub/test-curl-7564-4d114a"
Sep 16 12:26:22.775308 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:22.775284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mwt5\" (UniqueName: \"kubernetes.io/projected/ded73ea2-f332-466e-b0cc-68f10fe061ae-kube-api-access-8mwt5\") pod \"test-curl-7564-4d114a\" (UID: \"ded73ea2-f332-466e-b0cc-68f10fe061ae\") " pod="opendatahub/test-curl-7564-4d114a"
Sep 16 12:26:22.793274 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:22.793247    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-4d114a"
Sep 16 12:26:23.067878 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:23.067842    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-4d114a"]
Sep 16 12:26:23.069210 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:26:23.069176    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podded73ea2_f332_466e_b0cc_68f10fe061ae.slice/crio-9d8fabd568299351699affb62a0315aebd6846526e8efcf3ba0fc85a8061000a WatchSource:0}: Error finding container 9d8fabd568299351699affb62a0315aebd6846526e8efcf3ba0fc85a8061000a: Status 404 returned error can't find the container with id 9d8fabd568299351699affb62a0315aebd6846526e8efcf3ba0fc85a8061000a
Sep 16 12:26:23.537492 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:23.537449    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-4d114a" event={"ID":"ded73ea2-f332-466e-b0cc-68f10fe061ae","Type":"ContainerStarted","Data":"0324ccbad8351a6ca35f2051113c34f4d89e945743b9b334258f62fbfb181401"}
Sep 16 12:26:23.537945 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:23.537505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-4d114a" event={"ID":"ded73ea2-f332-466e-b0cc-68f10fe061ae","Type":"ContainerStarted","Data":"9d8fabd568299351699affb62a0315aebd6846526e8efcf3ba0fc85a8061000a"}
Sep 16 12:26:23.555004 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:23.554947    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7564-4d114a" podStartSLOduration=1.554931901 podStartE2EDuration="1.554931901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:23.552570086 +0000 UTC m=+1542.803107639" watchObservedRunningTime="2026-09-16 12:26:23.554931901 +0000 UTC m=+1542.805469450"
Sep 16 12:26:24.166921 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:24.166885    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-4d114a"]
Sep 16 12:26:24.172228 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:24.172199    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-4d114a"]
Sep 16 12:26:24.287160 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:24.287077    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-7220f1"]
Sep 16 12:26:24.287455 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:24.287443    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded73ea2-f332-466e-b0cc-68f10fe061ae" containerName="test-curl-7564-4d114a"
Sep 16 12:26:24.287505 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:24.287458    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded73ea2-f332-466e-b0cc-68f10fe061ae" containerName="test-curl-7564-4d114a"
Sep 16 12:26:24.287541 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:24.287528    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ded73ea2-f332-466e-b0cc-68f10fe061ae" containerName="test-curl-7564-4d114a"
Sep 16 12:26:24.290569 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:24.290553    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-7220f1"
Sep 16 12:26:24.299848 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:24.299810    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-7220f1"]
Sep 16 12:26:24.483376 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:24.483344    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp579\" (UniqueName: \"kubernetes.io/projected/56587056-4534-4eac-93b9-b5907f3448ef-kube-api-access-bp579\") pod \"test-curl-7564-7220f1\" (UID: \"56587056-4534-4eac-93b9-b5907f3448ef\") " pod="opendatahub/test-curl-7564-7220f1"
Sep 16 12:26:24.543087 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:24.543003    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7564-4d114a" container="test-curl-7564-4d114a"
Sep 16 12:26:24.584827 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:24.584794    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp579\" (UniqueName: \"kubernetes.io/projected/56587056-4534-4eac-93b9-b5907f3448ef-kube-api-access-bp579\") pod \"test-curl-7564-7220f1\" (UID: \"56587056-4534-4eac-93b9-b5907f3448ef\") " pod="opendatahub/test-curl-7564-7220f1"
Sep 16 12:26:24.593387 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:24.593352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp579\" (UniqueName: \"kubernetes.io/projected/56587056-4534-4eac-93b9-b5907f3448ef-kube-api-access-bp579\") pod \"test-curl-7564-7220f1\" (UID: \"56587056-4534-4eac-93b9-b5907f3448ef\") " pod="opendatahub/test-curl-7564-7220f1"
Sep 16 12:26:24.601252 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:24.601227    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-7220f1"
Sep 16 12:26:24.871746 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:24.871703    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-7220f1"]
Sep 16 12:26:24.872848 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:26:24.872821    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56587056_4534_4eac_93b9_b5907f3448ef.slice/crio-962688b7976da5ebaa52949b84e5cf97b7f2c49c7cb48a1fe9d8ba019d7ce9ce WatchSource:0}: Error finding container 962688b7976da5ebaa52949b84e5cf97b7f2c49c7cb48a1fe9d8ba019d7ce9ce: Status 404 returned error can't find the container with id 962688b7976da5ebaa52949b84e5cf97b7f2c49c7cb48a1fe9d8ba019d7ce9ce
Sep 16 12:26:25.550357 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:25.550308    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-7220f1" event={"ID":"56587056-4534-4eac-93b9-b5907f3448ef","Type":"ContainerStarted","Data":"d5cebacfc52b00946629ad73b38607236a50ec2a62f905f3fc8cb9ed9c81934b"}
Sep 16 12:26:25.550814 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:25.550364    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-7220f1" event={"ID":"56587056-4534-4eac-93b9-b5907f3448ef","Type":"ContainerStarted","Data":"962688b7976da5ebaa52949b84e5cf97b7f2c49c7cb48a1fe9d8ba019d7ce9ce"}
Sep 16 12:26:25.550814 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:25.550385    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7564-4d114a" podUID="ded73ea2-f332-466e-b0cc-68f10fe061ae" containerName="test-curl-7564-4d114a" containerID="cri-o://0324ccbad8351a6ca35f2051113c34f4d89e945743b9b334258f62fbfb181401" gracePeriod=2
Sep 16 12:26:25.565790 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:25.565726    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7564-7220f1" podStartSLOduration=1.565708501 podStartE2EDuration="1.565708501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:25.563801386 +0000 UTC m=+1544.814338935" watchObservedRunningTime="2026-09-16 12:26:25.565708501 +0000 UTC m=+1544.816246050"
Sep 16 12:26:26.189469 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:26.189433    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-7220f1"]
Sep 16 12:26:26.193500 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:26.193473    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-7220f1"]
Sep 16 12:26:26.313861 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:26.313770    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-5e11a6"]
Sep 16 12:26:26.314137 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:26.314124    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56587056-4534-4eac-93b9-b5907f3448ef" containerName="test-curl-7564-7220f1"
Sep 16 12:26:26.314183 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:26.314139    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="56587056-4534-4eac-93b9-b5907f3448ef" containerName="test-curl-7564-7220f1"
Sep 16 12:26:26.314226 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:26.314217    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="56587056-4534-4eac-93b9-b5907f3448ef" containerName="test-curl-7564-7220f1"
Sep 16 12:26:26.317572 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:26.317541    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-5e11a6"
Sep 16 12:26:26.323142 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:26.323114    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-5e11a6"]
Sep 16 12:26:26.401887 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:26.401854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zqsc\" (UniqueName: \"kubernetes.io/projected/78abb444-56a2-406e-998e-a6267d8daf40-kube-api-access-8zqsc\") pod \"test-curl-7564-5e11a6\" (UID: \"78abb444-56a2-406e-998e-a6267d8daf40\") " pod="opendatahub/test-curl-7564-5e11a6"
Sep 16 12:26:26.502757 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:26.502689    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zqsc\" (UniqueName: \"kubernetes.io/projected/78abb444-56a2-406e-998e-a6267d8daf40-kube-api-access-8zqsc\") pod \"test-curl-7564-5e11a6\" (UID: \"78abb444-56a2-406e-998e-a6267d8daf40\") " pod="opendatahub/test-curl-7564-5e11a6"
Sep 16 12:26:26.511380 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:26.511348    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zqsc\" (UniqueName: \"kubernetes.io/projected/78abb444-56a2-406e-998e-a6267d8daf40-kube-api-access-8zqsc\") pod \"test-curl-7564-5e11a6\" (UID: \"78abb444-56a2-406e-998e-a6267d8daf40\") " pod="opendatahub/test-curl-7564-5e11a6"
Sep 16 12:26:26.555958 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:26.555930    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7564-7220f1" container="test-curl-7564-7220f1"
Sep 16 12:26:26.628937 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:26.628901    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-5e11a6"
Sep 16 12:26:26.909850 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:26.909818    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-5e11a6"]
Sep 16 12:26:26.910598 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:26:26.910572    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78abb444_56a2_406e_998e_a6267d8daf40.slice/crio-33f0a94956ec07d3f454e7f043f7b4540a2484103ea56c57d1ae2085c06f9f5b WatchSource:0}: Error finding container 33f0a94956ec07d3f454e7f043f7b4540a2484103ea56c57d1ae2085c06f9f5b: Status 404 returned error can't find the container with id 33f0a94956ec07d3f454e7f043f7b4540a2484103ea56c57d1ae2085c06f9f5b
Sep 16 12:26:27.564058 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:27.564004    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-5e11a6" event={"ID":"78abb444-56a2-406e-998e-a6267d8daf40","Type":"ContainerStarted","Data":"269c0e2b1728bf354cb6449bdd074d9122d32b38585cdc68d73b26844ffe1ff8"}
Sep 16 12:26:27.564444 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:27.564066    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-5e11a6" event={"ID":"78abb444-56a2-406e-998e-a6267d8daf40","Type":"ContainerStarted","Data":"33f0a94956ec07d3f454e7f043f7b4540a2484103ea56c57d1ae2085c06f9f5b"}
Sep 16 12:26:27.564444 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:27.564092    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7564-7220f1" podUID="56587056-4534-4eac-93b9-b5907f3448ef" containerName="test-curl-7564-7220f1" containerID="cri-o://d5cebacfc52b00946629ad73b38607236a50ec2a62f905f3fc8cb9ed9c81934b" gracePeriod=2
Sep 16 12:26:27.582427 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:27.582377    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7564-5e11a6" podStartSLOduration=1.582359659 podStartE2EDuration="1.582359659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:27.581970005 +0000 UTC m=+1546.832507555" watchObservedRunningTime="2026-09-16 12:26:27.582359659 +0000 UTC m=+1546.832897209"
Sep 16 12:26:27.723209 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:27.723179    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-4d114a"
Sep 16 12:26:27.815952 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:27.815850    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mwt5\" (UniqueName: \"kubernetes.io/projected/ded73ea2-f332-466e-b0cc-68f10fe061ae-kube-api-access-8mwt5\") pod \"ded73ea2-f332-466e-b0cc-68f10fe061ae\" (UID: \"ded73ea2-f332-466e-b0cc-68f10fe061ae\") "
Sep 16 12:26:27.818329 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:27.818293    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded73ea2-f332-466e-b0cc-68f10fe061ae-kube-api-access-8mwt5" (OuterVolumeSpecName: "kube-api-access-8mwt5") pod "ded73ea2-f332-466e-b0cc-68f10fe061ae" (UID: "ded73ea2-f332-466e-b0cc-68f10fe061ae"). InnerVolumeSpecName "kube-api-access-8mwt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:26:27.916985 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:27.916946    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mwt5\" (UniqueName: \"kubernetes.io/projected/ded73ea2-f332-466e-b0cc-68f10fe061ae-kube-api-access-8mwt5\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:26:28.207790 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.207745    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-5e11a6"]
Sep 16 12:26:28.212399 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.212367    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-5e11a6"]
Sep 16 12:26:28.369976 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.369942    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-206cc0"]
Sep 16 12:26:28.370303 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.370290    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78abb444-56a2-406e-998e-a6267d8daf40" containerName="test-curl-7564-5e11a6"
Sep 16 12:26:28.370350 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.370305    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="78abb444-56a2-406e-998e-a6267d8daf40" containerName="test-curl-7564-5e11a6"
Sep 16 12:26:28.370386 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.370370    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="78abb444-56a2-406e-998e-a6267d8daf40" containerName="test-curl-7564-5e11a6"
Sep 16 12:26:28.373374 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.373359    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-206cc0"
Sep 16 12:26:28.386709 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.386682    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-206cc0"]
Sep 16 12:26:28.421156 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.421121    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26tr2\" (UniqueName: \"kubernetes.io/projected/c8080fa5-feac-4812-b62c-2e32c4c649b0-kube-api-access-26tr2\") pod \"test-curl-7564-206cc0\" (UID: \"c8080fa5-feac-4812-b62c-2e32c4c649b0\") " pod="opendatahub/test-curl-7564-206cc0"
Sep 16 12:26:28.522280 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.522186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26tr2\" (UniqueName: \"kubernetes.io/projected/c8080fa5-feac-4812-b62c-2e32c4c649b0-kube-api-access-26tr2\") pod \"test-curl-7564-206cc0\" (UID: \"c8080fa5-feac-4812-b62c-2e32c4c649b0\") " pod="opendatahub/test-curl-7564-206cc0"
Sep 16 12:26:28.530747 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.530710    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26tr2\" (UniqueName: \"kubernetes.io/projected/c8080fa5-feac-4812-b62c-2e32c4c649b0-kube-api-access-26tr2\") pod \"test-curl-7564-206cc0\" (UID: \"c8080fa5-feac-4812-b62c-2e32c4c649b0\") " pod="opendatahub/test-curl-7564-206cc0"
Sep 16 12:26:28.571032 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.570995    2582 generic.go:356] "Generic (PLEG): container finished" podID="ded73ea2-f332-466e-b0cc-68f10fe061ae" containerID="0324ccbad8351a6ca35f2051113c34f4d89e945743b9b334258f62fbfb181401" exitCode=137
Sep 16 12:26:28.571446 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.571049    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-4d114a"
Sep 16 12:26:28.571446 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.571097    2582 scope.go:117] "RemoveContainer" containerID="0324ccbad8351a6ca35f2051113c34f4d89e945743b9b334258f62fbfb181401"
Sep 16 12:26:28.571446 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:28.571392    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7564-5e11a6" container="test-curl-7564-5e11a6"
Sep 16 12:26:28.581195 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.581176    2582 scope.go:117] "RemoveContainer" containerID="0324ccbad8351a6ca35f2051113c34f4d89e945743b9b334258f62fbfb181401"
Sep 16 12:26:28.581476 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:28.581459    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0324ccbad8351a6ca35f2051113c34f4d89e945743b9b334258f62fbfb181401\": container with ID starting with 0324ccbad8351a6ca35f2051113c34f4d89e945743b9b334258f62fbfb181401 not found: ID does not exist" containerID="0324ccbad8351a6ca35f2051113c34f4d89e945743b9b334258f62fbfb181401"
Sep 16 12:26:28.581520 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.581491    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0324ccbad8351a6ca35f2051113c34f4d89e945743b9b334258f62fbfb181401"} err="failed to get container status \"0324ccbad8351a6ca35f2051113c34f4d89e945743b9b334258f62fbfb181401\": rpc error: code = NotFound desc = could not find container \"0324ccbad8351a6ca35f2051113c34f4d89e945743b9b334258f62fbfb181401\": container with ID starting with 0324ccbad8351a6ca35f2051113c34f4d89e945743b9b334258f62fbfb181401 not found: ID does not exist"
Sep 16 12:26:28.582569 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:28.582547    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-4d114a\" is forbidden: User \"system:node:ip-10-0-140-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-71.ec2.internal' and this object" podUID="ded73ea2-f332-466e-b0cc-68f10fe061ae" pod="opendatahub/test-curl-7564-4d114a"
Sep 16 12:26:28.683629 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.683597    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-206cc0"
Sep 16 12:26:28.957290 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:28.957253    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-206cc0"]
Sep 16 12:26:28.958406 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:26:28.958377    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8080fa5_feac_4812_b62c_2e32c4c649b0.slice/crio-2d4d6a583c68aa3bf7e035a8022df3402061b03c34e7bf3cbf27cea646ebc507 WatchSource:0}: Error finding container 2d4d6a583c68aa3bf7e035a8022df3402061b03c34e7bf3cbf27cea646ebc507: Status 404 returned error can't find the container with id 2d4d6a583c68aa3bf7e035a8022df3402061b03c34e7bf3cbf27cea646ebc507
Sep 16 12:26:29.212955 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.212864    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ded73ea2-f332-466e-b0cc-68f10fe061ae" path="/var/lib/kubelet/pods/ded73ea2-f332-466e-b0cc-68f10fe061ae/volumes"
Sep 16 12:26:29.579426 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.579118    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-206cc0" event={"ID":"c8080fa5-feac-4812-b62c-2e32c4c649b0","Type":"ContainerStarted","Data":"2433797768c5c137e663198c3bc3a3f9f87d3a3e345893610966acc6b9db1c05"}
Sep 16 12:26:29.579426 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.579165    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-206cc0" event={"ID":"c8080fa5-feac-4812-b62c-2e32c4c649b0","Type":"ContainerStarted","Data":"2d4d6a583c68aa3bf7e035a8022df3402061b03c34e7bf3cbf27cea646ebc507"}
Sep 16 12:26:29.580441 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.580401    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7564-5e11a6" podUID="78abb444-56a2-406e-998e-a6267d8daf40" containerName="test-curl-7564-5e11a6" containerID="cri-o://269c0e2b1728bf354cb6449bdd074d9122d32b38585cdc68d73b26844ffe1ff8" gracePeriod=2
Sep 16 12:26:29.599045 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.598995    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7564-206cc0" podStartSLOduration=1.598978521 podStartE2EDuration="1.598978521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:29.59714562 +0000 UTC m=+1548.847683172" watchObservedRunningTime="2026-09-16 12:26:29.598978521 +0000 UTC m=+1548.849516071"
Sep 16 12:26:29.733507 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.733470    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"]
Sep 16 12:26:29.737966 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.737939    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:26:29.740679 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.740657    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-72eee5-serving-cert\""
Sep 16 12:26:29.760955 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.760924    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"]
Sep 16 12:26:29.782888 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.782863    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-7220f1"
Sep 16 12:26:29.834676 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.834649    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp579\" (UniqueName: \"kubernetes.io/projected/56587056-4534-4eac-93b9-b5907f3448ef-kube-api-access-bp579\") pod \"56587056-4534-4eac-93b9-b5907f3448ef\" (UID: \"56587056-4534-4eac-93b9-b5907f3448ef\") "
Sep 16 12:26:29.834875 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.834783    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btv7s\" (UniqueName: \"kubernetes.io/projected/7989b85b-a710-40ad-9b0b-6ab2de094f14-kube-api-access-btv7s\") pod \"maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg\" (UID: \"7989b85b-a710-40ad-9b0b-6ab2de094f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:26:29.834875 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.834816    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7989b85b-a710-40ad-9b0b-6ab2de094f14-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg\" (UID: \"7989b85b-a710-40ad-9b0b-6ab2de094f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:26:29.834875 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.834848    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7989b85b-a710-40ad-9b0b-6ab2de094f14-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg\" (UID: \"7989b85b-a710-40ad-9b0b-6ab2de094f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:26:29.835067 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.834931    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7989b85b-a710-40ad-9b0b-6ab2de094f14-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg\" (UID: \"7989b85b-a710-40ad-9b0b-6ab2de094f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:26:29.836835 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.836807    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56587056-4534-4eac-93b9-b5907f3448ef-kube-api-access-bp579" (OuterVolumeSpecName: "kube-api-access-bp579") pod "56587056-4534-4eac-93b9-b5907f3448ef" (UID: "56587056-4534-4eac-93b9-b5907f3448ef"). InnerVolumeSpecName "kube-api-access-bp579". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:26:29.935622 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.935585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btv7s\" (UniqueName: \"kubernetes.io/projected/7989b85b-a710-40ad-9b0b-6ab2de094f14-kube-api-access-btv7s\") pod \"maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg\" (UID: \"7989b85b-a710-40ad-9b0b-6ab2de094f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:26:29.935622 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.935628    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7989b85b-a710-40ad-9b0b-6ab2de094f14-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg\" (UID: \"7989b85b-a710-40ad-9b0b-6ab2de094f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:26:29.935915 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.935866    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7989b85b-a710-40ad-9b0b-6ab2de094f14-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg\" (UID: \"7989b85b-a710-40ad-9b0b-6ab2de094f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:26:29.935976 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.935917    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7989b85b-a710-40ad-9b0b-6ab2de094f14-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg\" (UID: \"7989b85b-a710-40ad-9b0b-6ab2de094f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:26:29.936026 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.936013    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp579\" (UniqueName: \"kubernetes.io/projected/56587056-4534-4eac-93b9-b5907f3448ef-kube-api-access-bp579\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:26:29.936647 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.936549    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7989b85b-a710-40ad-9b0b-6ab2de094f14-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg\" (UID: \"7989b85b-a710-40ad-9b0b-6ab2de094f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:26:29.938224 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.938205    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7989b85b-a710-40ad-9b0b-6ab2de094f14-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg\" (UID: \"7989b85b-a710-40ad-9b0b-6ab2de094f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:26:29.938322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.938270    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7989b85b-a710-40ad-9b0b-6ab2de094f14-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg\" (UID: \"7989b85b-a710-40ad-9b0b-6ab2de094f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:26:29.944350 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:29.944326    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btv7s\" (UniqueName: \"kubernetes.io/projected/7989b85b-a710-40ad-9b0b-6ab2de094f14-kube-api-access-btv7s\") pod \"maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg\" (UID: \"7989b85b-a710-40ad-9b0b-6ab2de094f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:26:30.054504 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:30.054470    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:26:30.229163 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:30.229125    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-206cc0"]
Sep 16 12:26:30.233537 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:30.233504    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-206cc0"]
Sep 16 12:26:30.337707 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:30.337584    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"]
Sep 16 12:26:30.339886 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:26:30.339858    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7989b85b_a710_40ad_9b0b_6ab2de094f14.slice/crio-e1270b9738c0bc085330c8a4b9d2323d88dd67d2ea2281383cf968c4fd030a81 WatchSource:0}: Error finding container e1270b9738c0bc085330c8a4b9d2323d88dd67d2ea2281383cf968c4fd030a81: Status 404 returned error can't find the container with id e1270b9738c0bc085330c8a4b9d2323d88dd67d2ea2281383cf968c4fd030a81
Sep 16 12:26:30.587384 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:30.587353    2582 generic.go:356] "Generic (PLEG): container finished" podID="56587056-4534-4eac-93b9-b5907f3448ef" containerID="d5cebacfc52b00946629ad73b38607236a50ec2a62f905f3fc8cb9ed9c81934b" exitCode=137
Sep 16 12:26:30.587828 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:30.587408    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-7220f1"
Sep 16 12:26:30.587828 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:30.587453    2582 scope.go:117] "RemoveContainer" containerID="d5cebacfc52b00946629ad73b38607236a50ec2a62f905f3fc8cb9ed9c81934b"
Sep 16 12:26:30.588873 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:30.588853    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg" event={"ID":"7989b85b-a710-40ad-9b0b-6ab2de094f14","Type":"ContainerStarted","Data":"e1270b9738c0bc085330c8a4b9d2323d88dd67d2ea2281383cf968c4fd030a81"}
Sep 16 12:26:30.589088 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:30.589067    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7564-206cc0" container="test-curl-7564-206cc0"
Sep 16 12:26:30.599311 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:30.599295    2582 scope.go:117] "RemoveContainer" containerID="d5cebacfc52b00946629ad73b38607236a50ec2a62f905f3fc8cb9ed9c81934b"
Sep 16 12:26:30.599590 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:30.599574    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5cebacfc52b00946629ad73b38607236a50ec2a62f905f3fc8cb9ed9c81934b\": container with ID starting with d5cebacfc52b00946629ad73b38607236a50ec2a62f905f3fc8cb9ed9c81934b not found: ID does not exist" containerID="d5cebacfc52b00946629ad73b38607236a50ec2a62f905f3fc8cb9ed9c81934b"
Sep 16 12:26:30.599637 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:30.599597    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5cebacfc52b00946629ad73b38607236a50ec2a62f905f3fc8cb9ed9c81934b"} err="failed to get container status \"d5cebacfc52b00946629ad73b38607236a50ec2a62f905f3fc8cb9ed9c81934b\": rpc error: code = NotFound desc = could not find container \"d5cebacfc52b00946629ad73b38607236a50ec2a62f905f3fc8cb9ed9c81934b\": container with ID starting with d5cebacfc52b00946629ad73b38607236a50ec2a62f905f3fc8cb9ed9c81934b not found: ID does not exist"
Sep 16 12:26:30.601360 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:30.601338    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-7220f1\" is forbidden: User \"system:node:ip-10-0-140-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-71.ec2.internal' and this object" podUID="56587056-4534-4eac-93b9-b5907f3448ef" pod="opendatahub/test-curl-7564-7220f1"
Sep 16 12:26:31.211438 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:31.211341    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-7220f1\" is forbidden: User \"system:node:ip-10-0-140-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-71.ec2.internal' and this object" podUID="56587056-4534-4eac-93b9-b5907f3448ef" pod="opendatahub/test-curl-7564-7220f1"
Sep 16 12:26:31.213672 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:31.213649    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56587056-4534-4eac-93b9-b5907f3448ef" path="/var/lib/kubelet/pods/56587056-4534-4eac-93b9-b5907f3448ef/volumes"
Sep 16 12:26:31.597683 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:31.597644    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg" event={"ID":"7989b85b-a710-40ad-9b0b-6ab2de094f14","Type":"ContainerStarted","Data":"58d85d2be73c94358485398cbbf720d8c68fa829fa87902bd729bb0c33801f2e"}
Sep 16 12:26:31.598139 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:31.597791    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:26:31.598139 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:31.597882    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7564-206cc0" podUID="c8080fa5-feac-4812-b62c-2e32c4c649b0" containerName="test-curl-7564-206cc0" containerID="cri-o://2433797768c5c137e663198c3bc3a3f9f87d3a3e345893610966acc6b9db1c05" gracePeriod=2
Sep 16 12:26:31.617283 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:31.617230    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg" podStartSLOduration=2.102755691 podStartE2EDuration="2.617213951s" totalImagesPullingTime="514.45826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:31.614301589 +0000 UTC m=+1550.864839139" watchObservedRunningTime="2026-09-16 12:26:31.617213951 +0000 UTC m=+1550.867751508"
Sep 16 12:26:31.768825 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:31.768796    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-5e11a6"
Sep 16 12:26:31.852065 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:31.851961    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zqsc\" (UniqueName: \"kubernetes.io/projected/78abb444-56a2-406e-998e-a6267d8daf40-kube-api-access-8zqsc\") pod \"78abb444-56a2-406e-998e-a6267d8daf40\" (UID: \"78abb444-56a2-406e-998e-a6267d8daf40\") "
Sep 16 12:26:31.854283 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:31.854255    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78abb444-56a2-406e-998e-a6267d8daf40-kube-api-access-8zqsc" (OuterVolumeSpecName: "kube-api-access-8zqsc") pod "78abb444-56a2-406e-998e-a6267d8daf40" (UID: "78abb444-56a2-406e-998e-a6267d8daf40"). InnerVolumeSpecName "kube-api-access-8zqsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:26:31.953475 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:31.953438    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zqsc\" (UniqueName: \"kubernetes.io/projected/78abb444-56a2-406e-998e-a6267d8daf40-kube-api-access-8zqsc\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:26:32.605036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:32.604994    2582 generic.go:356] "Generic (PLEG): container finished" podID="78abb444-56a2-406e-998e-a6267d8daf40" containerID="269c0e2b1728bf354cb6449bdd074d9122d32b38585cdc68d73b26844ffe1ff8" exitCode=137
Sep 16 12:26:32.605036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:32.605034    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-5e11a6"
Sep 16 12:26:32.605624 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:32.605094    2582 scope.go:117] "RemoveContainer" containerID="269c0e2b1728bf354cb6449bdd074d9122d32b38585cdc68d73b26844ffe1ff8"
Sep 16 12:26:32.614604 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:32.614584    2582 scope.go:117] "RemoveContainer" containerID="269c0e2b1728bf354cb6449bdd074d9122d32b38585cdc68d73b26844ffe1ff8"
Sep 16 12:26:32.614920 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:32.614899    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"269c0e2b1728bf354cb6449bdd074d9122d32b38585cdc68d73b26844ffe1ff8\": container with ID starting with 269c0e2b1728bf354cb6449bdd074d9122d32b38585cdc68d73b26844ffe1ff8 not found: ID does not exist" containerID="269c0e2b1728bf354cb6449bdd074d9122d32b38585cdc68d73b26844ffe1ff8"
Sep 16 12:26:32.614968 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:32.614930    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"269c0e2b1728bf354cb6449bdd074d9122d32b38585cdc68d73b26844ffe1ff8"} err="failed to get container status \"269c0e2b1728bf354cb6449bdd074d9122d32b38585cdc68d73b26844ffe1ff8\": rpc error: code = NotFound desc = could not find container \"269c0e2b1728bf354cb6449bdd074d9122d32b38585cdc68d73b26844ffe1ff8\": container with ID starting with 269c0e2b1728bf354cb6449bdd074d9122d32b38585cdc68d73b26844ffe1ff8 not found: ID does not exist"
Sep 16 12:26:32.615364 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:32.615342    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-5e11a6\" is forbidden: User \"system:node:ip-10-0-140-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-71.ec2.internal' and this object" podUID="78abb444-56a2-406e-998e-a6267d8daf40" pod="opendatahub/test-curl-7564-5e11a6"
Sep 16 12:26:33.212196 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:33.212166    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78abb444-56a2-406e-998e-a6267d8daf40" path="/var/lib/kubelet/pods/78abb444-56a2-406e-998e-a6267d8daf40/volumes"
Sep 16 12:26:33.782125 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:33.782094    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-206cc0"
Sep 16 12:26:33.870457 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:33.870371    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26tr2\" (UniqueName: \"kubernetes.io/projected/c8080fa5-feac-4812-b62c-2e32c4c649b0-kube-api-access-26tr2\") pod \"c8080fa5-feac-4812-b62c-2e32c4c649b0\" (UID: \"c8080fa5-feac-4812-b62c-2e32c4c649b0\") "
Sep 16 12:26:33.872812 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:33.872782    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8080fa5-feac-4812-b62c-2e32c4c649b0-kube-api-access-26tr2" (OuterVolumeSpecName: "kube-api-access-26tr2") pod "c8080fa5-feac-4812-b62c-2e32c4c649b0" (UID: "c8080fa5-feac-4812-b62c-2e32c4c649b0"). InnerVolumeSpecName "kube-api-access-26tr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:26:33.972178 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:33.972141    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26tr2\" (UniqueName: \"kubernetes.io/projected/c8080fa5-feac-4812-b62c-2e32c4c649b0-kube-api-access-26tr2\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:26:34.620180 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:34.620145    2582 generic.go:356] "Generic (PLEG): container finished" podID="c8080fa5-feac-4812-b62c-2e32c4c649b0" containerID="2433797768c5c137e663198c3bc3a3f9f87d3a3e345893610966acc6b9db1c05" exitCode=137
Sep 16 12:26:34.620396 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:34.620187    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-206cc0"
Sep 16 12:26:34.620396 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:34.620248    2582 scope.go:117] "RemoveContainer" containerID="2433797768c5c137e663198c3bc3a3f9f87d3a3e345893610966acc6b9db1c05"
Sep 16 12:26:34.629497 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:34.629479    2582 scope.go:117] "RemoveContainer" containerID="2433797768c5c137e663198c3bc3a3f9f87d3a3e345893610966acc6b9db1c05"
Sep 16 12:26:34.629799 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:34.629777    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2433797768c5c137e663198c3bc3a3f9f87d3a3e345893610966acc6b9db1c05\": container with ID starting with 2433797768c5c137e663198c3bc3a3f9f87d3a3e345893610966acc6b9db1c05 not found: ID does not exist" containerID="2433797768c5c137e663198c3bc3a3f9f87d3a3e345893610966acc6b9db1c05"
Sep 16 12:26:34.629854 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:34.629813    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2433797768c5c137e663198c3bc3a3f9f87d3a3e345893610966acc6b9db1c05"} err="failed to get container status \"2433797768c5c137e663198c3bc3a3f9f87d3a3e345893610966acc6b9db1c05\": rpc error: code = NotFound desc = could not find container \"2433797768c5c137e663198c3bc3a3f9f87d3a3e345893610966acc6b9db1c05\": container with ID starting with 2433797768c5c137e663198c3bc3a3f9f87d3a3e345893610966acc6b9db1c05 not found: ID does not exist"
Sep 16 12:26:34.630284 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:34.630261    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-206cc0\" is forbidden: User \"system:node:ip-10-0-140-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-71.ec2.internal' and this object" podUID="c8080fa5-feac-4812-b62c-2e32c4c649b0" pod="opendatahub/test-curl-7564-206cc0"
Sep 16 12:26:35.212842 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:35.212805    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8080fa5-feac-4812-b62c-2e32c4c649b0" path="/var/lib/kubelet/pods/c8080fa5-feac-4812-b62c-2e32c4c649b0/volumes"
Sep 16 12:26:36.990580 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:36.990543    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"]
Sep 16 12:26:36.991185 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:36.991162    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8080fa5-feac-4812-b62c-2e32c4c649b0" containerName="test-curl-7564-206cc0"
Sep 16 12:26:36.991230 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:36.991192    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8080fa5-feac-4812-b62c-2e32c4c649b0" containerName="test-curl-7564-206cc0"
Sep 16 12:26:36.991323 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:36.991311    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8080fa5-feac-4812-b62c-2e32c4c649b0" containerName="test-curl-7564-206cc0"
Sep 16 12:26:36.996309 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:36.996283    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:26:36.998795 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:36.998768    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b1650816-serving-cert\""
Sep 16 12:26:37.003198 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:37.003173    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"]
Sep 16 12:26:37.099892 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:37.099848    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxgl8\" (UniqueName: \"kubernetes.io/projected/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-kube-api-access-jxgl8\") pod \"maas-api-e2e-mt-b1650816-75f86896c5-wfqsr\" (UID: \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:26:37.100111 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:37.099913    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-metrics-certs\") pod \"maas-api-e2e-mt-b1650816-75f86896c5-wfqsr\" (UID: \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:26:37.100111 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:37.099992    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-maas-api-tls\") pod \"maas-api-e2e-mt-b1650816-75f86896c5-wfqsr\" (UID: \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:26:37.100111 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:37.100030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-openshift-service-ca\") pod \"maas-api-e2e-mt-b1650816-75f86896c5-wfqsr\" (UID: \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:26:37.201339 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:37.201281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-maas-api-tls\") pod \"maas-api-e2e-mt-b1650816-75f86896c5-wfqsr\" (UID: \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:26:37.201339 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:37.201352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-openshift-service-ca\") pod \"maas-api-e2e-mt-b1650816-75f86896c5-wfqsr\" (UID: \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:26:37.201636 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:37.201443    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-serving-cert: secret "maas-api-e2e-mt-b1650816-serving-cert" not found
Sep 16 12:26:37.201636 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:37.201453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxgl8\" (UniqueName: \"kubernetes.io/projected/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-kube-api-access-jxgl8\") pod \"maas-api-e2e-mt-b1650816-75f86896c5-wfqsr\" (UID: \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:26:37.201636 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:26:37.201516    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-maas-api-tls podName:65e0b5bf-139b-49db-95b2-501f0d3ce7a5 nodeName:}" failed. No retries permitted until 2026-09-16 12:26:37.701492792 +0000 UTC m=+1556.952030338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-maas-api-tls") pod "maas-api-e2e-mt-b1650816-75f86896c5-wfqsr" (UID: "65e0b5bf-139b-49db-95b2-501f0d3ce7a5") : secret "maas-api-e2e-mt-b1650816-serving-cert" not found
Sep 16 12:26:37.201636 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:37.201535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-metrics-certs\") pod \"maas-api-e2e-mt-b1650816-75f86896c5-wfqsr\" (UID: \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:26:37.202189 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:37.202162    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-openshift-service-ca\") pod \"maas-api-e2e-mt-b1650816-75f86896c5-wfqsr\" (UID: \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:26:37.204376 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:37.204351    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-metrics-certs\") pod \"maas-api-e2e-mt-b1650816-75f86896c5-wfqsr\" (UID: \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:26:37.211425 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:37.211392    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxgl8\" (UniqueName: \"kubernetes.io/projected/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-kube-api-access-jxgl8\") pod \"maas-api-e2e-mt-b1650816-75f86896c5-wfqsr\" (UID: \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:26:37.611405 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:37.611374    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:26:37.708203 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:37.708164    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-maas-api-tls\") pod \"maas-api-e2e-mt-b1650816-75f86896c5-wfqsr\" (UID: \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:26:37.710877 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:37.710842    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-maas-api-tls\") pod \"maas-api-e2e-mt-b1650816-75f86896c5-wfqsr\" (UID: \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:26:37.910864 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:37.910771    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:26:38.183505 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:38.183479    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"]
Sep 16 12:26:38.185043 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:26:38.185013    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e0b5bf_139b_49db_95b2_501f0d3ce7a5.slice/crio-55a86d721b4dd5c61728df4b9677631970de955a1408b32546e73f2e2b772fc5 WatchSource:0}: Error finding container 55a86d721b4dd5c61728df4b9677631970de955a1408b32546e73f2e2b772fc5: Status 404 returned error can't find the container with id 55a86d721b4dd5c61728df4b9677631970de955a1408b32546e73f2e2b772fc5
Sep 16 12:26:38.646693 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:38.646658    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr" event={"ID":"65e0b5bf-139b-49db-95b2-501f0d3ce7a5","Type":"ContainerStarted","Data":"240ec23fbe4bcf8138e8e3637d627656cfb9040492c3bfc8b5a25dbd5371c702"}
Sep 16 12:26:38.646889 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:38.646713    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr" event={"ID":"65e0b5bf-139b-49db-95b2-501f0d3ce7a5","Type":"ContainerStarted","Data":"55a86d721b4dd5c61728df4b9677631970de955a1408b32546e73f2e2b772fc5"}
Sep 16 12:26:38.646889 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:38.646788    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:26:38.665367 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:38.665322    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr" podStartSLOduration=2.323960718 podStartE2EDuration="2.665305028s" totalImagesPullingTime="341.34431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:38.664078052 +0000 UTC m=+1557.914615601" watchObservedRunningTime="2026-09-16 12:26:38.665305028 +0000 UTC m=+1557.915842577"
Sep 16 12:26:44.658054 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:44.658024    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:26:52.392845 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.392809    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"]
Sep 16 12:26:52.398005 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.397985    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.401720 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.401690    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-37953c\"/\"kube-root-ca.crt\""
Sep 16 12:26:52.401917 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.401723    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-37953c\"/\"openshift-service-ca.crt\""
Sep 16 12:26:52.401917 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.401690    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-37953c\"/\"default-dockercfg-24szp\""
Sep 16 12:26:52.401917 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.401803    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-37953c\"/\"noten-model-37953c-kserve-self-signed-certs\""
Sep 16 12:26:52.417256 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.417222    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"]
Sep 16 12:26:52.553978 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.553940    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-model-cache\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.554184 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.553996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-tmp-dir\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.554184 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.554035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955a1968-fa89-4acc-8e45-99425cfea5d2-tls-certs\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.554184 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.554058    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz6zm\" (UniqueName: \"kubernetes.io/projected/955a1968-fa89-4acc-8e45-99425cfea5d2-kube-api-access-mz6zm\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.554184 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.554086    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-home\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.554351 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.554193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-dshm\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.655141 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.655054    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955a1968-fa89-4acc-8e45-99425cfea5d2-tls-certs\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.655141 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.655095    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz6zm\" (UniqueName: \"kubernetes.io/projected/955a1968-fa89-4acc-8e45-99425cfea5d2-kube-api-access-mz6zm\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.655141 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.655124    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-home\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.655411 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.655188    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-dshm\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.655411 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.655243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-model-cache\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.655579 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.655547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-tmp-dir\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.655728 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.655623    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-home\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.655728 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.655643    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-model-cache\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.655869 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.655845    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-tmp-dir\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.657761 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.657721    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-dshm\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.658024 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.658004    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955a1968-fa89-4acc-8e45-99425cfea5d2-tls-certs\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.665690 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.665667    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz6zm\" (UniqueName: \"kubernetes.io/projected/955a1968-fa89-4acc-8e45-99425cfea5d2-kube-api-access-mz6zm\") pod \"noten-model-37953c-kserve-79cbc4f667-zsd7m\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") " pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.710119 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.710074    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:26:52.984556 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:52.984529    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"]
Sep 16 12:26:52.985767 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:26:52.985708    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod955a1968_fa89_4acc_8e45_99425cfea5d2.slice/crio-08b510ec5220decc413fdd8c7dfb4dbf2fc44d5ee041f17668bd8043acd3f5a5 WatchSource:0}: Error finding container 08b510ec5220decc413fdd8c7dfb4dbf2fc44d5ee041f17668bd8043acd3f5a5: Status 404 returned error can't find the container with id 08b510ec5220decc413fdd8c7dfb4dbf2fc44d5ee041f17668bd8043acd3f5a5
Sep 16 12:26:53.732920 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:53.732877    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m" event={"ID":"955a1968-fa89-4acc-8e45-99425cfea5d2","Type":"ContainerStarted","Data":"195b5cc055f4d51102141c4b736547e10e83312be4a93c02849ce518d741dafd"}
Sep 16 12:26:53.733326 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:53.732933    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m" event={"ID":"955a1968-fa89-4acc-8e45-99425cfea5d2","Type":"ContainerStarted","Data":"08b510ec5220decc413fdd8c7dfb4dbf2fc44d5ee041f17668bd8043acd3f5a5"}
Sep 16 12:26:53.752024 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:53.751971    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m" podStartSLOduration=1.751954799 podStartE2EDuration="1.751954799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:53.749551352 +0000 UTC m=+1573.000088901" watchObservedRunningTime="2026-09-16 12:26:53.751954799 +0000 UTC m=+1573.002492348"
Sep 16 12:26:58.689184 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:58.689152    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"]
Sep 16 12:26:59.993928 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:59.993885    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"]
Sep 16 12:26:59.998335 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:26:59.998306    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:27:00.001012 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:00.000988    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-3be0c5-serving-cert\""
Sep 16 12:27:00.010773 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:00.010727    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"]
Sep 16 12:27:00.128793 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:00.128749    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dde19631-63ed-4833-8a70-e3bd95007396-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2\" (UID: \"dde19631-63ed-4833-8a70-e3bd95007396\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:27:00.128793 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:00.128798    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q64xw\" (UniqueName: \"kubernetes.io/projected/dde19631-63ed-4833-8a70-e3bd95007396-kube-api-access-q64xw\") pod \"maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2\" (UID: \"dde19631-63ed-4833-8a70-e3bd95007396\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:27:00.129025 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:00.128918    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dde19631-63ed-4833-8a70-e3bd95007396-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2\" (UID: \"dde19631-63ed-4833-8a70-e3bd95007396\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:27:00.129025 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:00.128979    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dde19631-63ed-4833-8a70-e3bd95007396-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2\" (UID: \"dde19631-63ed-4833-8a70-e3bd95007396\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:27:00.229816 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:00.229774    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dde19631-63ed-4833-8a70-e3bd95007396-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2\" (UID: \"dde19631-63ed-4833-8a70-e3bd95007396\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:27:00.229994 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:00.229847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dde19631-63ed-4833-8a70-e3bd95007396-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2\" (UID: \"dde19631-63ed-4833-8a70-e3bd95007396\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:27:00.229994 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:00.229910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dde19631-63ed-4833-8a70-e3bd95007396-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2\" (UID: \"dde19631-63ed-4833-8a70-e3bd95007396\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:27:00.229994 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:00.229943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q64xw\" (UniqueName: \"kubernetes.io/projected/dde19631-63ed-4833-8a70-e3bd95007396-kube-api-access-q64xw\") pod \"maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2\" (UID: \"dde19631-63ed-4833-8a70-e3bd95007396\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:27:00.230178 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:27:00.230071    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-serving-cert: secret "maas-api-e2e-shared-b-w3-3be0c5-serving-cert" not found
Sep 16 12:27:00.230178 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:27:00.230144    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dde19631-63ed-4833-8a70-e3bd95007396-maas-api-tls podName:dde19631-63ed-4833-8a70-e3bd95007396 nodeName:}" failed. No retries permitted until 2026-09-16 12:27:00.73012364 +0000 UTC m=+1579.980661167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dde19631-63ed-4833-8a70-e3bd95007396-maas-api-tls") pod "maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2" (UID: "dde19631-63ed-4833-8a70-e3bd95007396") : secret "maas-api-e2e-shared-b-w3-3be0c5-serving-cert" not found
Sep 16 12:27:00.230557 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:00.230533    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dde19631-63ed-4833-8a70-e3bd95007396-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2\" (UID: \"dde19631-63ed-4833-8a70-e3bd95007396\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:27:00.232450 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:00.232428    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dde19631-63ed-4833-8a70-e3bd95007396-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2\" (UID: \"dde19631-63ed-4833-8a70-e3bd95007396\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:27:00.242161 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:00.242138    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q64xw\" (UniqueName: \"kubernetes.io/projected/dde19631-63ed-4833-8a70-e3bd95007396-kube-api-access-q64xw\") pod \"maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2\" (UID: \"dde19631-63ed-4833-8a70-e3bd95007396\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:27:00.735338 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:00.735302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dde19631-63ed-4833-8a70-e3bd95007396-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2\" (UID: \"dde19631-63ed-4833-8a70-e3bd95007396\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:27:00.738037 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:00.738013    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dde19631-63ed-4833-8a70-e3bd95007396-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2\" (UID: \"dde19631-63ed-4833-8a70-e3bd95007396\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:27:00.912361 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:00.912309    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:27:01.212610 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:27:01.212573    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde19631_63ed_4833_8a70_e3bd95007396.slice/crio-89296db3cafd12ff0a926ca89b057e3a5fc3369546c233d226ea6e89338e99e7 WatchSource:0}: Error finding container 89296db3cafd12ff0a926ca89b057e3a5fc3369546c233d226ea6e89338e99e7: Status 404 returned error can't find the container with id 89296db3cafd12ff0a926ca89b057e3a5fc3369546c233d226ea6e89338e99e7
Sep 16 12:27:01.213599 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:01.213577    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"]
Sep 16 12:27:01.781285 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:01.781235    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2" event={"ID":"dde19631-63ed-4833-8a70-e3bd95007396","Type":"ContainerStarted","Data":"af2b07b389817ceb1183066a348ce4f7beeae71b5bb27640dc8c806577b2ea3b"}
Sep 16 12:27:01.781474 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:01.781308    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:27:01.781474 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:01.781334    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2" event={"ID":"dde19631-63ed-4833-8a70-e3bd95007396","Type":"ContainerStarted","Data":"89296db3cafd12ff0a926ca89b057e3a5fc3369546c233d226ea6e89338e99e7"}
Sep 16 12:27:01.803713 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:01.803649    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2" podStartSLOduration=2.43100585 podStartE2EDuration="2.803629953s" totalImagesPullingTime="372.624103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:27:01.801719133 +0000 UTC m=+1581.052256705" watchObservedRunningTime="2026-09-16 12:27:01.803629953 +0000 UTC m=+1581.054167546"
Sep 16 12:27:02.710953 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:02.710906    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:27:07.792752 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:07.792698    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:27:13.709807 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:13.709727    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m" podUID="955a1968-fa89-4acc-8e45-99425cfea5d2" containerName="main" containerID="cri-o://195b5cc055f4d51102141c4b736547e10e83312be4a93c02849ce518d741dafd" gracePeriod=45
Sep 16 12:27:13.853973 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:13.853930    2582 generic.go:356] "Generic (PLEG): container finished" podID="955a1968-fa89-4acc-8e45-99425cfea5d2" containerID="195b5cc055f4d51102141c4b736547e10e83312be4a93c02849ce518d741dafd" exitCode=0
Sep 16 12:27:13.854155 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:13.853994    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m" event={"ID":"955a1968-fa89-4acc-8e45-99425cfea5d2","Type":"ContainerDied","Data":"195b5cc055f4d51102141c4b736547e10e83312be4a93c02849ce518d741dafd"}
Sep 16 12:27:13.969415 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:13.969354    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:27:14.065893 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.065865    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955a1968-fa89-4acc-8e45-99425cfea5d2-tls-certs\") pod \"955a1968-fa89-4acc-8e45-99425cfea5d2\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") "
Sep 16 12:27:14.066067 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.065900    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-model-cache\") pod \"955a1968-fa89-4acc-8e45-99425cfea5d2\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") "
Sep 16 12:27:14.066067 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.065941    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-dshm\") pod \"955a1968-fa89-4acc-8e45-99425cfea5d2\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") "
Sep 16 12:27:14.066067 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.066030    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz6zm\" (UniqueName: \"kubernetes.io/projected/955a1968-fa89-4acc-8e45-99425cfea5d2-kube-api-access-mz6zm\") pod \"955a1968-fa89-4acc-8e45-99425cfea5d2\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") "
Sep 16 12:27:14.066217 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.066070    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-tmp-dir\") pod \"955a1968-fa89-4acc-8e45-99425cfea5d2\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") "
Sep 16 12:27:14.066217 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.066114    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-home\") pod \"955a1968-fa89-4acc-8e45-99425cfea5d2\" (UID: \"955a1968-fa89-4acc-8e45-99425cfea5d2\") "
Sep 16 12:27:14.066217 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.066187    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "955a1968-fa89-4acc-8e45-99425cfea5d2" (UID: "955a1968-fa89-4acc-8e45-99425cfea5d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:14.066437 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.066405    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "955a1968-fa89-4acc-8e45-99425cfea5d2" (UID: "955a1968-fa89-4acc-8e45-99425cfea5d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:14.066525 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.066443    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-home" (OuterVolumeSpecName: "home") pod "955a1968-fa89-4acc-8e45-99425cfea5d2" (UID: "955a1968-fa89-4acc-8e45-99425cfea5d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:14.066525 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.066453    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:27:14.068456 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.068428    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-dshm" (OuterVolumeSpecName: "dshm") pod "955a1968-fa89-4acc-8e45-99425cfea5d2" (UID: "955a1968-fa89-4acc-8e45-99425cfea5d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:14.068456 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.068434    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955a1968-fa89-4acc-8e45-99425cfea5d2-kube-api-access-mz6zm" (OuterVolumeSpecName: "kube-api-access-mz6zm") pod "955a1968-fa89-4acc-8e45-99425cfea5d2" (UID: "955a1968-fa89-4acc-8e45-99425cfea5d2"). InnerVolumeSpecName "kube-api-access-mz6zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:14.068625 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.068461    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/955a1968-fa89-4acc-8e45-99425cfea5d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "955a1968-fa89-4acc-8e45-99425cfea5d2" (UID: "955a1968-fa89-4acc-8e45-99425cfea5d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:14.166966 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.166918    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:27:14.166966 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.166954    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955a1968-fa89-4acc-8e45-99425cfea5d2-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:27:14.166966 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.166964    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:27:14.166966 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.166973    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz6zm\" (UniqueName: \"kubernetes.io/projected/955a1968-fa89-4acc-8e45-99425cfea5d2-kube-api-access-mz6zm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:27:14.166966 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.166981    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955a1968-fa89-4acc-8e45-99425cfea5d2-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:27:14.860830 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.860801    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"
Sep 16 12:27:14.861318 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.860834    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m" event={"ID":"955a1968-fa89-4acc-8e45-99425cfea5d2","Type":"ContainerDied","Data":"08b510ec5220decc413fdd8c7dfb4dbf2fc44d5ee041f17668bd8043acd3f5a5"}
Sep 16 12:27:14.861318 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.860890    2582 scope.go:117] "RemoveContainer" containerID="195b5cc055f4d51102141c4b736547e10e83312be4a93c02849ce518d741dafd"
Sep 16 12:27:14.889485 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.889461    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"]
Sep 16 12:27:14.900616 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:14.900589    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-37953c/noten-model-37953c-kserve-79cbc4f667-zsd7m"]
Sep 16 12:27:15.213234 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:15.213147    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="955a1968-fa89-4acc-8e45-99425cfea5d2" path="/var/lib/kubelet/pods/955a1968-fa89-4acc-8e45-99425cfea5d2/volumes"
Sep 16 12:27:19.386343 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.386310    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"]
Sep 16 12:27:19.386800 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.386758    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="955a1968-fa89-4acc-8e45-99425cfea5d2" containerName="main"
Sep 16 12:27:19.386800 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.386775    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="955a1968-fa89-4acc-8e45-99425cfea5d2" containerName="main"
Sep 16 12:27:19.386888 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.386849    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="955a1968-fa89-4acc-8e45-99425cfea5d2" containerName="main"
Sep 16 12:27:19.391263 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.391244    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.395563 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.395534    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b1650816\"/\"default-dockercfg-fwrq8\""
Sep 16 12:27:19.395563 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.395553    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b1650816\"/\"openshift-service-ca.crt\""
Sep 16 12:27:19.395825 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.395563    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b1650816\"/\"kube-root-ca.crt\""
Sep 16 12:27:19.395825 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.395534    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b1650816\"/\"e2e-lifecycle-model-b1650816-kserve-self-signed-certs\""
Sep 16 12:27:19.400165 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.399788    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"]
Sep 16 12:27:19.522554 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.522513    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-dshm\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.522756 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.522575    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-667c6\" (UniqueName: \"kubernetes.io/projected/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-kube-api-access-667c6\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.522756 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.522678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-home\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.522756 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.522725    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-tmp-dir\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.522873 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.522777    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-tls-certs\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.522873 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.522807    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-model-cache\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.624088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.624050    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-home\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.624303 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.624094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-tmp-dir\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.624303 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.624114    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-tls-certs\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.624303 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.624249    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-model-cache\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.624488 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.624362    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-dshm\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.624488 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.624445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667c6\" (UniqueName: \"kubernetes.io/projected/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-kube-api-access-667c6\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.624596 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.624535    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-tmp-dir\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.624596 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.624556    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-home\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.624663 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.624608    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-model-cache\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.626691 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.626665    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-dshm\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.626892 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.626875    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-tls-certs\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.633799 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.633776    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-667c6\" (UniqueName: \"kubernetes.io/projected/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-kube-api-access-667c6\") pod \"e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") " pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.702818 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.702706    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:19.976636 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:19.976607    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"]
Sep 16 12:27:19.977906 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:27:19.977879    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podada09f37_c99b_4d05_8cc9_5fc438a6f2b4.slice/crio-0a3c91783153ff7deb1fa2cfa8d8d7e81898a230bc109b0454b65e803ab9d8cf WatchSource:0}: Error finding container 0a3c91783153ff7deb1fa2cfa8d8d7e81898a230bc109b0454b65e803ab9d8cf: Status 404 returned error can't find the container with id 0a3c91783153ff7deb1fa2cfa8d8d7e81898a230bc109b0454b65e803ab9d8cf
Sep 16 12:27:20.898538 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:20.898499    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj" event={"ID":"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4","Type":"ContainerStarted","Data":"1ca55d3dc175e6728757e6f0f72948ac1b5e2f670b3b64e40a615db6ec3dd2af"}
Sep 16 12:27:20.898988 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:20.898549    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj" event={"ID":"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4","Type":"ContainerStarted","Data":"0a3c91783153ff7deb1fa2cfa8d8d7e81898a230bc109b0454b65e803ab9d8cf"}
Sep 16 12:27:20.917018 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:20.916958    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj" podStartSLOduration=1.916939765 podStartE2EDuration="1.916939765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:27:20.916048331 +0000 UTC m=+1600.166585882" watchObservedRunningTime="2026-09-16 12:27:20.916939765 +0000 UTC m=+1600.167477314"
Sep 16 12:27:29.703862 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:29.703821    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:29.704306 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:29.703879    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:29.716425 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:29.716396    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:29.966356 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:29.966265    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:27:36.105034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.104956    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"]
Sep 16 12:27:36.109788 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.109725    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.112800 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.112774    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7a4d5c\"/\"distinct-w4-7a4d5c-kserve-self-signed-certs\""
Sep 16 12:27:36.121276 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.121250    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"]
Sep 16 12:27:36.186300 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.186263    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9bb324-3182-43ff-b96b-03d578c69153-tls-certs\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.186300 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.186304    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-dshm\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.186519 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.186339    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-model-cache\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.186519 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.186363    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-tmp-dir\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.186519 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.186386    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhgnh\" (UniqueName: \"kubernetes.io/projected/5a9bb324-3182-43ff-b96b-03d578c69153-kube-api-access-nhgnh\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.186519 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.186446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-home\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.288006 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.287968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9bb324-3182-43ff-b96b-03d578c69153-tls-certs\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.288006 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.288006    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-dshm\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.288259 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.288041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-model-cache\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.288259 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.288069    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-tmp-dir\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.288259 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.288095    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhgnh\" (UniqueName: \"kubernetes.io/projected/5a9bb324-3182-43ff-b96b-03d578c69153-kube-api-access-nhgnh\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.288259 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.288160    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-home\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.288525 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.288503    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-model-cache\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.288588 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.288560    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-tmp-dir\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.288625 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.288588    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-home\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.290400 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.290374    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-dshm\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.290655 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.290637    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9bb324-3182-43ff-b96b-03d578c69153-tls-certs\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.296450 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.296426    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhgnh\" (UniqueName: \"kubernetes.io/projected/5a9bb324-3182-43ff-b96b-03d578c69153-kube-api-access-nhgnh\") pod \"distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") " pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.421860 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.421719    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:36.694879 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.694846    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"]
Sep 16 12:27:36.696782 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:27:36.696726    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a9bb324_3182_43ff_b96b_03d578c69153.slice/crio-10e966cac32273c1c7153bfb7fc83bb1c586d2f77b921056005e1efe905351f2 WatchSource:0}: Error finding container 10e966cac32273c1c7153bfb7fc83bb1c586d2f77b921056005e1efe905351f2: Status 404 returned error can't find the container with id 10e966cac32273c1c7153bfb7fc83bb1c586d2f77b921056005e1efe905351f2
Sep 16 12:27:36.999893 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.999769    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm" event={"ID":"5a9bb324-3182-43ff-b96b-03d578c69153","Type":"ContainerStarted","Data":"9670624a9a2021cae55ec54fa13b3522b4dcd3bac5232d76e7537965c7456991"}
Sep 16 12:27:36.999893 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:36.999843    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm" event={"ID":"5a9bb324-3182-43ff-b96b-03d578c69153","Type":"ContainerStarted","Data":"10e966cac32273c1c7153bfb7fc83bb1c586d2f77b921056005e1efe905351f2"}
Sep 16 12:27:37.019302 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:37.019249    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm" podStartSLOduration=1.019232439 podStartE2EDuration="1.019232439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:27:37.018524543 +0000 UTC m=+1616.269062092" watchObservedRunningTime="2026-09-16 12:27:37.019232439 +0000 UTC m=+1616.269769987"
Sep 16 12:27:46.421934 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:46.421886    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:46.422382 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:46.421948    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:46.435369 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:46.435341    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:47.067282 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:47.067254    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:27:51.115410 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.115368    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"]
Sep 16 12:27:51.120904 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.120878    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.124127 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.124099    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-72eee5\"/\"openshift-service-ca.crt\""
Sep 16 12:27:51.124394 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.124234    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-72eee5\"/\"default-dockercfg-9d5h8\""
Sep 16 12:27:51.124826 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.124804    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-72eee5\"/\"auth-test-model-72eee5-kserve-self-signed-certs\""
Sep 16 12:27:51.125077 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.125063    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-72eee5\"/\"kube-root-ca.crt\""
Sep 16 12:27:51.126987 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.126966    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"]
Sep 16 12:27:51.235669 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.235633    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-model-cache\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.235896 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.235826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-dshm\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.235896 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.235867    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-home\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.236022 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.235957    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-tmp-dir\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.236022 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.235995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnwlk\" (UniqueName: \"kubernetes.io/projected/1b5f731c-b509-4f45-a119-e7d4e520ac63-kube-api-access-cnwlk\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.236133 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.236098    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b5f731c-b509-4f45-a119-e7d4e520ac63-tls-certs\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.336761 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.336697    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b5f731c-b509-4f45-a119-e7d4e520ac63-tls-certs\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.336961 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.336780    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-model-cache\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.336961 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.336826    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-dshm\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.336961 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.336841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-home\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.336961 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.336886    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-tmp-dir\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.336961 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.336914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnwlk\" (UniqueName: \"kubernetes.io/projected/1b5f731c-b509-4f45-a119-e7d4e520ac63-kube-api-access-cnwlk\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.337275 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.337248    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-model-cache\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.337328 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.337313    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-home\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.337365 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.337330    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-tmp-dir\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.339246 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.339223    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-dshm\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.339523 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.339503    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b5f731c-b509-4f45-a119-e7d4e520ac63-tls-certs\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.345754 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.345717    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnwlk\" (UniqueName: \"kubernetes.io/projected/1b5f731c-b509-4f45-a119-e7d4e520ac63-kube-api-access-cnwlk\") pod \"auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.432061 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.431956    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:27:51.706675 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:51.706641    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"]
Sep 16 12:27:51.708813 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:27:51.708776    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b5f731c_b509_4f45_a119_e7d4e520ac63.slice/crio-95b5304a39471bab346b86f9e05d9412db7e52e1fca3e36bf523870a6217c131 WatchSource:0}: Error finding container 95b5304a39471bab346b86f9e05d9412db7e52e1fca3e36bf523870a6217c131: Status 404 returned error can't find the container with id 95b5304a39471bab346b86f9e05d9412db7e52e1fca3e36bf523870a6217c131
Sep 16 12:27:52.089053 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:52.089018    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z" event={"ID":"1b5f731c-b509-4f45-a119-e7d4e520ac63","Type":"ContainerStarted","Data":"6a3990abe042e3ceb6efb6193076089f4c82e38f6adff50408d03ab2e22576d3"}
Sep 16 12:27:52.089053 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:52.089060    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z" event={"ID":"1b5f731c-b509-4f45-a119-e7d4e520ac63","Type":"ContainerStarted","Data":"95b5304a39471bab346b86f9e05d9412db7e52e1fca3e36bf523870a6217c131"}
Sep 16 12:27:52.107165 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:27:52.107108    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z" podStartSLOduration=1.107088391 podStartE2EDuration="1.107088391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:27:52.104893971 +0000 UTC m=+1631.355431569" watchObservedRunningTime="2026-09-16 12:27:52.107088391 +0000 UTC m=+1631.357625980"
Sep 16 12:28:01.432890 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:01.432849    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:28:01.432890 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:01.432896    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:28:01.445385 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:01.445352    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:28:02.157892 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:02.157857    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:28:15.599332 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:15.599298    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"]
Sep 16 12:28:15.599959 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:15.599560    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr" podUID="65e0b5bf-139b-49db-95b2-501f0d3ce7a5" containerName="maas-api" containerID="cri-o://240ec23fbe4bcf8138e8e3637d627656cfb9040492c3bfc8b5a25dbd5371c702" gracePeriod=30
Sep 16 12:28:15.861414 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:15.861351    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:28:15.992628 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:15.992595    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-openshift-service-ca\") pod \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\" (UID: \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\") "
Sep 16 12:28:15.992843 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:15.992644    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxgl8\" (UniqueName: \"kubernetes.io/projected/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-kube-api-access-jxgl8\") pod \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\" (UID: \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\") "
Sep 16 12:28:15.992843 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:15.992726    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-maas-api-tls\") pod \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\" (UID: \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\") "
Sep 16 12:28:15.992843 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:15.992797    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-metrics-certs\") pod \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\" (UID: \"65e0b5bf-139b-49db-95b2-501f0d3ce7a5\") "
Sep 16 12:28:15.993097 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:15.993058    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65e0b5bf-139b-49db-95b2-501f0d3ce7a5" (UID: "65e0b5bf-139b-49db-95b2-501f0d3ce7a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:28:15.995050 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:15.995013    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-kube-api-access-jxgl8" (OuterVolumeSpecName: "kube-api-access-jxgl8") pod "65e0b5bf-139b-49db-95b2-501f0d3ce7a5" (UID: "65e0b5bf-139b-49db-95b2-501f0d3ce7a5"). InnerVolumeSpecName "kube-api-access-jxgl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:15.995050 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:15.995038    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "65e0b5bf-139b-49db-95b2-501f0d3ce7a5" (UID: "65e0b5bf-139b-49db-95b2-501f0d3ce7a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:28:15.995198 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:15.995122    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65e0b5bf-139b-49db-95b2-501f0d3ce7a5" (UID: "65e0b5bf-139b-49db-95b2-501f0d3ce7a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:28:16.094446 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:16.094405    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:28:16.094446 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:16.094441    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:28:16.094446 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:16.094451    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:28:16.094689 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:16.094460    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxgl8\" (UniqueName: \"kubernetes.io/projected/65e0b5bf-139b-49db-95b2-501f0d3ce7a5-kube-api-access-jxgl8\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:28:16.228963 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:16.228874    2582 generic.go:356] "Generic (PLEG): container finished" podID="65e0b5bf-139b-49db-95b2-501f0d3ce7a5" containerID="240ec23fbe4bcf8138e8e3637d627656cfb9040492c3bfc8b5a25dbd5371c702" exitCode=0
Sep 16 12:28:16.228963 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:16.228924    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"
Sep 16 12:28:16.228963 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:16.228954    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr" event={"ID":"65e0b5bf-139b-49db-95b2-501f0d3ce7a5","Type":"ContainerDied","Data":"240ec23fbe4bcf8138e8e3637d627656cfb9040492c3bfc8b5a25dbd5371c702"}
Sep 16 12:28:16.229219 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:16.229008    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr" event={"ID":"65e0b5bf-139b-49db-95b2-501f0d3ce7a5","Type":"ContainerDied","Data":"55a86d721b4dd5c61728df4b9677631970de955a1408b32546e73f2e2b772fc5"}
Sep 16 12:28:16.229219 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:16.229030    2582 scope.go:117] "RemoveContainer" containerID="240ec23fbe4bcf8138e8e3637d627656cfb9040492c3bfc8b5a25dbd5371c702"
Sep 16 12:28:16.239403 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:16.239383    2582 scope.go:117] "RemoveContainer" containerID="240ec23fbe4bcf8138e8e3637d627656cfb9040492c3bfc8b5a25dbd5371c702"
Sep 16 12:28:16.239687 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:28:16.239665    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240ec23fbe4bcf8138e8e3637d627656cfb9040492c3bfc8b5a25dbd5371c702\": container with ID starting with 240ec23fbe4bcf8138e8e3637d627656cfb9040492c3bfc8b5a25dbd5371c702 not found: ID does not exist" containerID="240ec23fbe4bcf8138e8e3637d627656cfb9040492c3bfc8b5a25dbd5371c702"
Sep 16 12:28:16.239764 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:16.239698    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240ec23fbe4bcf8138e8e3637d627656cfb9040492c3bfc8b5a25dbd5371c702"} err="failed to get container status \"240ec23fbe4bcf8138e8e3637d627656cfb9040492c3bfc8b5a25dbd5371c702\": rpc error: code = NotFound desc = could not find container \"240ec23fbe4bcf8138e8e3637d627656cfb9040492c3bfc8b5a25dbd5371c702\": container with ID starting with 240ec23fbe4bcf8138e8e3637d627656cfb9040492c3bfc8b5a25dbd5371c702 not found: ID does not exist"
Sep 16 12:28:16.255495 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:16.255471    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"]
Sep 16 12:28:16.258035 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:16.258017    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1650816-75f86896c5-wfqsr"]
Sep 16 12:28:17.212811 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:17.212774    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65e0b5bf-139b-49db-95b2-501f0d3ce7a5" path="/var/lib/kubelet/pods/65e0b5bf-139b-49db-95b2-501f0d3ce7a5/volumes"
Sep 16 12:28:24.441018 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:24.440979    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"]
Sep 16 12:28:39.460940 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.460878    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj" podUID="ada09f37-c99b-4d05-8cc9-5fc438a6f2b4" containerName="main" containerID="cri-o://1ca55d3dc175e6728757e6f0f72948ac1b5e2f670b3b64e40a615db6ec3dd2af" gracePeriod=45
Sep 16 12:28:39.719603 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.719535    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:28:39.839458 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.839419    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-tmp-dir\") pod \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") "
Sep 16 12:28:39.839646 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.839499    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-model-cache\") pod \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") "
Sep 16 12:28:39.839646 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.839559    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-home\") pod \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") "
Sep 16 12:28:39.839646 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.839597    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-dshm\") pod \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") "
Sep 16 12:28:39.839646 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.839630    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-tls-certs\") pod \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") "
Sep 16 12:28:39.839919 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.839721    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "ada09f37-c99b-4d05-8cc9-5fc438a6f2b4" (UID: "ada09f37-c99b-4d05-8cc9-5fc438a6f2b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:39.839919 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.839766    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ada09f37-c99b-4d05-8cc9-5fc438a6f2b4" (UID: "ada09f37-c99b-4d05-8cc9-5fc438a6f2b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:39.839919 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.839761    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-667c6\" (UniqueName: \"kubernetes.io/projected/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-kube-api-access-667c6\") pod \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\" (UID: \"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4\") "
Sep 16 12:28:39.839919 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.839855    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-home" (OuterVolumeSpecName: "home") pod "ada09f37-c99b-4d05-8cc9-5fc438a6f2b4" (UID: "ada09f37-c99b-4d05-8cc9-5fc438a6f2b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:39.840197 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.840174    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:28:39.840277 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.840198    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:28:39.840277 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.840208    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:28:39.842098 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.842066    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-dshm" (OuterVolumeSpecName: "dshm") pod "ada09f37-c99b-4d05-8cc9-5fc438a6f2b4" (UID: "ada09f37-c99b-4d05-8cc9-5fc438a6f2b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:39.842205 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.842130    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ada09f37-c99b-4d05-8cc9-5fc438a6f2b4" (UID: "ada09f37-c99b-4d05-8cc9-5fc438a6f2b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:28:39.842205 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.842135    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-kube-api-access-667c6" (OuterVolumeSpecName: "kube-api-access-667c6") pod "ada09f37-c99b-4d05-8cc9-5fc438a6f2b4" (UID: "ada09f37-c99b-4d05-8cc9-5fc438a6f2b4"). InnerVolumeSpecName "kube-api-access-667c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:39.941151 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.941111    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-667c6\" (UniqueName: \"kubernetes.io/projected/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-kube-api-access-667c6\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:28:39.941151 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.941146    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:28:39.941151 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:39.941156    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:28:40.370595 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:40.370557    2582 generic.go:356] "Generic (PLEG): container finished" podID="ada09f37-c99b-4d05-8cc9-5fc438a6f2b4" containerID="1ca55d3dc175e6728757e6f0f72948ac1b5e2f670b3b64e40a615db6ec3dd2af" exitCode=0
Sep 16 12:28:40.370792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:40.370600    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"
Sep 16 12:28:40.370792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:40.370635    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj" event={"ID":"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4","Type":"ContainerDied","Data":"1ca55d3dc175e6728757e6f0f72948ac1b5e2f670b3b64e40a615db6ec3dd2af"}
Sep 16 12:28:40.370792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:40.370681    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj" event={"ID":"ada09f37-c99b-4d05-8cc9-5fc438a6f2b4","Type":"ContainerDied","Data":"0a3c91783153ff7deb1fa2cfa8d8d7e81898a230bc109b0454b65e803ab9d8cf"}
Sep 16 12:28:40.370792 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:40.370704    2582 scope.go:117] "RemoveContainer" containerID="1ca55d3dc175e6728757e6f0f72948ac1b5e2f670b3b64e40a615db6ec3dd2af"
Sep 16 12:28:40.381609 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:40.381590    2582 scope.go:117] "RemoveContainer" containerID="1ca55d3dc175e6728757e6f0f72948ac1b5e2f670b3b64e40a615db6ec3dd2af"
Sep 16 12:28:40.381980 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:28:40.381955    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca55d3dc175e6728757e6f0f72948ac1b5e2f670b3b64e40a615db6ec3dd2af\": container with ID starting with 1ca55d3dc175e6728757e6f0f72948ac1b5e2f670b3b64e40a615db6ec3dd2af not found: ID does not exist" containerID="1ca55d3dc175e6728757e6f0f72948ac1b5e2f670b3b64e40a615db6ec3dd2af"
Sep 16 12:28:40.382076 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:40.381994    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca55d3dc175e6728757e6f0f72948ac1b5e2f670b3b64e40a615db6ec3dd2af"} err="failed to get container status \"1ca55d3dc175e6728757e6f0f72948ac1b5e2f670b3b64e40a615db6ec3dd2af\": rpc error: code = NotFound desc = could not find container \"1ca55d3dc175e6728757e6f0f72948ac1b5e2f670b3b64e40a615db6ec3dd2af\": container with ID starting with 1ca55d3dc175e6728757e6f0f72948ac1b5e2f670b3b64e40a615db6ec3dd2af not found: ID does not exist"
Sep 16 12:28:40.392832 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:40.392807    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"]
Sep 16 12:28:40.396963 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:40.396937    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b1650816/e2e-lifecycle-model-b1650816-kserve-784ff5579-p7jcj"]
Sep 16 12:28:41.212802 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:28:41.212768    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ada09f37-c99b-4d05-8cc9-5fc438a6f2b4" path="/var/lib/kubelet/pods/ada09f37-c99b-4d05-8cc9-5fc438a6f2b4/volumes"
Sep 16 12:29:06.211244 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.211153    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"]
Sep 16 12:29:06.211838 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.211586    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65e0b5bf-139b-49db-95b2-501f0d3ce7a5" containerName="maas-api"
Sep 16 12:29:06.211838 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.211600    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="65e0b5bf-139b-49db-95b2-501f0d3ce7a5" containerName="maas-api"
Sep 16 12:29:06.211838 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.211618    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ada09f37-c99b-4d05-8cc9-5fc438a6f2b4" containerName="main"
Sep 16 12:29:06.211838 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.211623    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ada09f37-c99b-4d05-8cc9-5fc438a6f2b4" containerName="main"
Sep 16 12:29:06.211838 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.211695    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ada09f37-c99b-4d05-8cc9-5fc438a6f2b4" containerName="main"
Sep 16 12:29:06.211838 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.211703    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="65e0b5bf-139b-49db-95b2-501f0d3ce7a5" containerName="maas-api"
Sep 16 12:29:06.216282 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.216258    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.220344 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.220323    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3be0c5\"/\"kube-root-ca.crt\""
Sep 16 12:29:06.220499 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.220396    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3be0c5\"/\"openshift-service-ca.crt\""
Sep 16 12:29:06.220499 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.220451    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3be0c5\"/\"default-dockercfg-c4qmb\""
Sep 16 12:29:06.220499 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.220397    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3be0c5\"/\"auth-test-model-3be0c5-kserve-self-signed-certs\""
Sep 16 12:29:06.225574 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.225544    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"]
Sep 16 12:29:06.402710 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.402662    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-dshm\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.402957 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.402830    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-home\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.402957 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.402905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-tmp-dir\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.402957 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.402930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0741457e-9313-48d6-a2ce-0f12dd916445-tls-certs\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.403117 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.402980    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-model-cache\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.403117 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.403052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d57gs\" (UniqueName: \"kubernetes.io/projected/0741457e-9313-48d6-a2ce-0f12dd916445-kube-api-access-d57gs\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.503643 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.503551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-model-cache\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.503643 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.503593    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d57gs\" (UniqueName: \"kubernetes.io/projected/0741457e-9313-48d6-a2ce-0f12dd916445-kube-api-access-d57gs\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.503643 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.503630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-dshm\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.503997 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.503692    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-home\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.503997 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.503724    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-tmp-dir\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.503997 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.503777    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0741457e-9313-48d6-a2ce-0f12dd916445-tls-certs\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.504169 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.504041    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-model-cache\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.504169 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.504151    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-home\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.504252 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.504185    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-tmp-dir\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.506111 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.506092    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-dshm\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.506481 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.506462    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0741457e-9313-48d6-a2ce-0f12dd916445-tls-certs\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.512928 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.512901    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d57gs\" (UniqueName: \"kubernetes.io/projected/0741457e-9313-48d6-a2ce-0f12dd916445-kube-api-access-d57gs\") pod \"auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.527684 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.527648    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:06.802141 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:06.802116    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"]
Sep 16 12:29:06.803425 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:29:06.803401    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0741457e_9313_48d6_a2ce_0f12dd916445.slice/crio-a55528431cac922c6df24a61c0db9e710f46ac64ba2addb2a40d5fbeca282ff0 WatchSource:0}: Error finding container a55528431cac922c6df24a61c0db9e710f46ac64ba2addb2a40d5fbeca282ff0: Status 404 returned error can't find the container with id a55528431cac922c6df24a61c0db9e710f46ac64ba2addb2a40d5fbeca282ff0
Sep 16 12:29:07.522745 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:07.522692    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8" event={"ID":"0741457e-9313-48d6-a2ce-0f12dd916445","Type":"ContainerStarted","Data":"ab0f9e103eda50e270e4d3c8d44e0da13f31804a4a2cbfbb87b1c4842570cc08"}
Sep 16 12:29:07.523161 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:07.522752    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8" event={"ID":"0741457e-9313-48d6-a2ce-0f12dd916445","Type":"ContainerStarted","Data":"a55528431cac922c6df24a61c0db9e710f46ac64ba2addb2a40d5fbeca282ff0"}
Sep 16 12:29:07.539097 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:07.539046    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8" podStartSLOduration=1.539029297 podStartE2EDuration="1.539029297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:29:07.538318571 +0000 UTC m=+1706.788856121" watchObservedRunningTime="2026-09-16 12:29:07.539029297 +0000 UTC m=+1706.789566846"
Sep 16 12:29:16.528224 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:16.528176    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:16.528675 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:16.528248    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:16.541329 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:16.541304    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:29:16.587657 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:29:16.587631    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:30:24.138110 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:24.138075    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"]
Sep 16 12:30:24.142021 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:24.141996    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:30:24.144884 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:24.144857    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-835ec8-serving-cert\""
Sep 16 12:30:24.155227 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:24.155192    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"]
Sep 16 12:30:24.187191 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:24.187149    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-maas-api-tls\") pod \"maas-api-e2e-api-a-835ec8-69f7898945-7q6kp\" (UID: \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:30:24.187382 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:24.187263    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-metrics-certs\") pod \"maas-api-e2e-api-a-835ec8-69f7898945-7q6kp\" (UID: \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:30:24.187382 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:24.187307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26sfs\" (UniqueName: \"kubernetes.io/projected/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-kube-api-access-26sfs\") pod \"maas-api-e2e-api-a-835ec8-69f7898945-7q6kp\" (UID: \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:30:24.187382 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:24.187351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-openshift-service-ca\") pod \"maas-api-e2e-api-a-835ec8-69f7898945-7q6kp\" (UID: \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:30:24.287907 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:24.287867    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-openshift-service-ca\") pod \"maas-api-e2e-api-a-835ec8-69f7898945-7q6kp\" (UID: \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:30:24.288096 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:24.287975    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-maas-api-tls\") pod \"maas-api-e2e-api-a-835ec8-69f7898945-7q6kp\" (UID: \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:30:24.288096 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:24.288081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-metrics-certs\") pod \"maas-api-e2e-api-a-835ec8-69f7898945-7q6kp\" (UID: \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:30:24.288340 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:30:24.288319    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-serving-cert: secret "maas-api-e2e-api-a-835ec8-serving-cert" not found
Sep 16 12:30:24.288394 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:24.288346    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26sfs\" (UniqueName: \"kubernetes.io/projected/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-kube-api-access-26sfs\") pod \"maas-api-e2e-api-a-835ec8-69f7898945-7q6kp\" (UID: \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:30:24.288489 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:30:24.288471    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-maas-api-tls podName:a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59 nodeName:}" failed. No retries permitted until 2026-09-16 12:30:24.788448404 +0000 UTC m=+1784.038985931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-maas-api-tls") pod "maas-api-e2e-api-a-835ec8-69f7898945-7q6kp" (UID: "a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59") : secret "maas-api-e2e-api-a-835ec8-serving-cert" not found
Sep 16 12:30:24.288648 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:24.288512    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-openshift-service-ca\") pod \"maas-api-e2e-api-a-835ec8-69f7898945-7q6kp\" (UID: \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:30:24.290949 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:24.290926    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-metrics-certs\") pod \"maas-api-e2e-api-a-835ec8-69f7898945-7q6kp\" (UID: \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:30:24.297271 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:24.297237    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26sfs\" (UniqueName: \"kubernetes.io/projected/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-kube-api-access-26sfs\") pod \"maas-api-e2e-api-a-835ec8-69f7898945-7q6kp\" (UID: \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:30:24.794490 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:24.794456    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-maas-api-tls\") pod \"maas-api-e2e-api-a-835ec8-69f7898945-7q6kp\" (UID: \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:30:24.797266 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:24.797229    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-maas-api-tls\") pod \"maas-api-e2e-api-a-835ec8-69f7898945-7q6kp\" (UID: \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:30:25.062797 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:25.062691    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:30:25.337446 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:25.337417    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"]
Sep 16 12:30:25.338665 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:30:25.338639    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e76409_a2c4_4ece_bf9f_0aa0c8d4ca59.slice/crio-8e411e05dfcfe384c8595a6b277bdb81307878a523a3439b8579c7a7a382e989 WatchSource:0}: Error finding container 8e411e05dfcfe384c8595a6b277bdb81307878a523a3439b8579c7a7a382e989: Status 404 returned error can't find the container with id 8e411e05dfcfe384c8595a6b277bdb81307878a523a3439b8579c7a7a382e989
Sep 16 12:30:25.340172 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:25.340154    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:30:25.966756 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:25.966704    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp" event={"ID":"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59","Type":"ContainerStarted","Data":"f04ffa886b81e04fc955c4342147ff54d7439571933d3ce6a814a8d5bcf4add2"}
Sep 16 12:30:25.966945 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:25.966785    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp" event={"ID":"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59","Type":"ContainerStarted","Data":"8e411e05dfcfe384c8595a6b277bdb81307878a523a3439b8579c7a7a382e989"}
Sep 16 12:30:25.966945 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:25.966808    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:30:25.984605 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:25.984500    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp" podStartSLOduration=1.5947879280000001 podStartE2EDuration="1.984483229s" totalImagesPullingTime="389.695301ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:30:25.982528625 +0000 UTC m=+1785.233066215" watchObservedRunningTime="2026-09-16 12:30:25.984483229 +0000 UTC m=+1785.235020779"
Sep 16 12:30:31.978351 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:31.978316    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:30:59.009423 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.009380    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"]
Sep 16 12:30:59.018097 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.018070    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:30:59.020901 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.020875    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5c41db-serving-cert\""
Sep 16 12:30:59.023670 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.023642    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"]
Sep 16 12:30:59.038962 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.038933    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b63944-f9c6-4f4b-b562-e2ab196c34ac-maas-api-tls\") pod \"maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk\" (UID: \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:30:59.038962 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.038966    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b63944-f9c6-4f4b-b562-e2ab196c34ac-metrics-certs\") pod \"maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk\" (UID: \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:30:59.039168 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.038999    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkrfw\" (UniqueName: \"kubernetes.io/projected/89b63944-f9c6-4f4b-b562-e2ab196c34ac-kube-api-access-zkrfw\") pod \"maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk\" (UID: \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:30:59.039168 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.039111    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b63944-f9c6-4f4b-b562-e2ab196c34ac-openshift-service-ca\") pod \"maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk\" (UID: \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:30:59.139903 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.139861    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b63944-f9c6-4f4b-b562-e2ab196c34ac-openshift-service-ca\") pod \"maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk\" (UID: \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:30:59.140097 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.139949    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b63944-f9c6-4f4b-b562-e2ab196c34ac-maas-api-tls\") pod \"maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk\" (UID: \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:30:59.140097 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.139969    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b63944-f9c6-4f4b-b562-e2ab196c34ac-metrics-certs\") pod \"maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk\" (UID: \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:30:59.140097 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.140000    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrfw\" (UniqueName: \"kubernetes.io/projected/89b63944-f9c6-4f4b-b562-e2ab196c34ac-kube-api-access-zkrfw\") pod \"maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk\" (UID: \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:30:59.140208 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:30:59.140102    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-serving-cert: secret "maas-api-e2e-api-b-5c41db-serving-cert" not found
Sep 16 12:30:59.140208 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:30:59.140173    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b63944-f9c6-4f4b-b562-e2ab196c34ac-maas-api-tls podName:89b63944-f9c6-4f4b-b562-e2ab196c34ac nodeName:}" failed. No retries permitted until 2026-09-16 12:30:59.640153601 +0000 UTC m=+1818.890691127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89b63944-f9c6-4f4b-b562-e2ab196c34ac-maas-api-tls") pod "maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk" (UID: "89b63944-f9c6-4f4b-b562-e2ab196c34ac") : secret "maas-api-e2e-api-b-5c41db-serving-cert" not found
Sep 16 12:30:59.140535 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.140510    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b63944-f9c6-4f4b-b562-e2ab196c34ac-openshift-service-ca\") pod \"maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk\" (UID: \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:30:59.142703 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.142679    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b63944-f9c6-4f4b-b562-e2ab196c34ac-metrics-certs\") pod \"maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk\" (UID: \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:30:59.148906 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.148883    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkrfw\" (UniqueName: \"kubernetes.io/projected/89b63944-f9c6-4f4b-b562-e2ab196c34ac-kube-api-access-zkrfw\") pod \"maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk\" (UID: \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:30:59.645209 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.645173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b63944-f9c6-4f4b-b562-e2ab196c34ac-maas-api-tls\") pod \"maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk\" (UID: \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:30:59.647942 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.647916    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b63944-f9c6-4f4b-b562-e2ab196c34ac-maas-api-tls\") pod \"maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk\" (UID: \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:30:59.931694 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:30:59.931607    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:31:00.210366 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:00.210341    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"]
Sep 16 12:31:00.211856 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:31:00.211827    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b63944_f9c6_4f4b_b562_e2ab196c34ac.slice/crio-bc078bb5f6b367dcd2a9bed20e89489af26067884c1585e92eeffed12fb23ba9 WatchSource:0}: Error finding container bc078bb5f6b367dcd2a9bed20e89489af26067884c1585e92eeffed12fb23ba9: Status 404 returned error can't find the container with id bc078bb5f6b367dcd2a9bed20e89489af26067884c1585e92eeffed12fb23ba9
Sep 16 12:31:01.167593 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:01.167550    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk" event={"ID":"89b63944-f9c6-4f4b-b562-e2ab196c34ac","Type":"ContainerStarted","Data":"f6b3409e780d0d81184be5d170e77c90e069ca7e1c92dc5216ee72e6db2f5025"}
Sep 16 12:31:01.167834 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:01.167624    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk" event={"ID":"89b63944-f9c6-4f4b-b562-e2ab196c34ac","Type":"ContainerStarted","Data":"bc078bb5f6b367dcd2a9bed20e89489af26067884c1585e92eeffed12fb23ba9"}
Sep 16 12:31:01.167834 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:01.167757    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:31:01.185213 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:01.185159    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk" podStartSLOduration=2.836407011 podStartE2EDuration="3.185139276s" totalImagesPullingTime="348.732265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:31:01.183730831 +0000 UTC m=+1820.434268380" watchObservedRunningTime="2026-09-16 12:31:01.185139276 +0000 UTC m=+1820.435676824"
Sep 16 12:31:07.179103 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:07.179074    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:31:27.011300 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:27.011260    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"]
Sep 16 12:31:27.027087 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:27.027052    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"]
Sep 16 12:31:27.038204 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:27.038169    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"]
Sep 16 12:31:39.418291 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:39.418253    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"]
Sep 16 12:31:39.419153 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:39.419118    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849" podUID="4a048b31-681c-4da8-bdfd-7fe0d837fd67" containerName="maas-api" containerID="cri-o://997c8555c0db6c0beef10df5e2d0e7bad12b9a1db3bfcdc6f0cee42652f5beab" gracePeriod=30
Sep 16 12:31:39.672268 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:39.672204    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:31:39.753150 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:39.753111    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a048b31-681c-4da8-bdfd-7fe0d837fd67-maas-api-tls\") pod \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\" (UID: \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\") "
Sep 16 12:31:39.753326 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:39.753170    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a048b31-681c-4da8-bdfd-7fe0d837fd67-openshift-service-ca\") pod \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\" (UID: \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\") "
Sep 16 12:31:39.753326 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:39.753235    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a048b31-681c-4da8-bdfd-7fe0d837fd67-metrics-certs\") pod \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\" (UID: \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\") "
Sep 16 12:31:39.753326 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:39.753256    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vdtc\" (UniqueName: \"kubernetes.io/projected/4a048b31-681c-4da8-bdfd-7fe0d837fd67-kube-api-access-2vdtc\") pod \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\" (UID: \"4a048b31-681c-4da8-bdfd-7fe0d837fd67\") "
Sep 16 12:31:39.753520 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:39.753490    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a048b31-681c-4da8-bdfd-7fe0d837fd67-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a048b31-681c-4da8-bdfd-7fe0d837fd67" (UID: "4a048b31-681c-4da8-bdfd-7fe0d837fd67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:31:39.755536 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:39.755513    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a048b31-681c-4da8-bdfd-7fe0d837fd67-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a048b31-681c-4da8-bdfd-7fe0d837fd67" (UID: "4a048b31-681c-4da8-bdfd-7fe0d837fd67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:31:39.755682 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:39.755666    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a048b31-681c-4da8-bdfd-7fe0d837fd67-kube-api-access-2vdtc" (OuterVolumeSpecName: "kube-api-access-2vdtc") pod "4a048b31-681c-4da8-bdfd-7fe0d837fd67" (UID: "4a048b31-681c-4da8-bdfd-7fe0d837fd67"). InnerVolumeSpecName "kube-api-access-2vdtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:31:39.755794 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:39.755687    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a048b31-681c-4da8-bdfd-7fe0d837fd67-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4a048b31-681c-4da8-bdfd-7fe0d837fd67" (UID: "4a048b31-681c-4da8-bdfd-7fe0d837fd67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:31:39.854153 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:39.854119    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a048b31-681c-4da8-bdfd-7fe0d837fd67-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:39.854153 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:39.854146    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a048b31-681c-4da8-bdfd-7fe0d837fd67-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:39.854153 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:39.854156    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a048b31-681c-4da8-bdfd-7fe0d837fd67-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:39.854153 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:39.854164    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vdtc\" (UniqueName: \"kubernetes.io/projected/4a048b31-681c-4da8-bdfd-7fe0d837fd67-kube-api-access-2vdtc\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:40.410478 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:40.410442    2582 generic.go:356] "Generic (PLEG): container finished" podID="4a048b31-681c-4da8-bdfd-7fe0d837fd67" containerID="997c8555c0db6c0beef10df5e2d0e7bad12b9a1db3bfcdc6f0cee42652f5beab" exitCode=0
Sep 16 12:31:40.410671 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:40.410504    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"
Sep 16 12:31:40.410671 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:40.410521    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849" event={"ID":"4a048b31-681c-4da8-bdfd-7fe0d837fd67","Type":"ContainerDied","Data":"997c8555c0db6c0beef10df5e2d0e7bad12b9a1db3bfcdc6f0cee42652f5beab"}
Sep 16 12:31:40.410671 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:40.410601    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849" event={"ID":"4a048b31-681c-4da8-bdfd-7fe0d837fd67","Type":"ContainerDied","Data":"a5c352f1dfa030cb663f629b5391b3cb9ad8ea2ae5559c60351d19d62e69cbea"}
Sep 16 12:31:40.410671 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:40.410623    2582 scope.go:117] "RemoveContainer" containerID="997c8555c0db6c0beef10df5e2d0e7bad12b9a1db3bfcdc6f0cee42652f5beab"
Sep 16 12:31:40.421707 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:40.421535    2582 scope.go:117] "RemoveContainer" containerID="997c8555c0db6c0beef10df5e2d0e7bad12b9a1db3bfcdc6f0cee42652f5beab"
Sep 16 12:31:40.421972 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:31:40.421866    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997c8555c0db6c0beef10df5e2d0e7bad12b9a1db3bfcdc6f0cee42652f5beab\": container with ID starting with 997c8555c0db6c0beef10df5e2d0e7bad12b9a1db3bfcdc6f0cee42652f5beab not found: ID does not exist" containerID="997c8555c0db6c0beef10df5e2d0e7bad12b9a1db3bfcdc6f0cee42652f5beab"
Sep 16 12:31:40.421972 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:40.421893    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997c8555c0db6c0beef10df5e2d0e7bad12b9a1db3bfcdc6f0cee42652f5beab"} err="failed to get container status \"997c8555c0db6c0beef10df5e2d0e7bad12b9a1db3bfcdc6f0cee42652f5beab\": rpc error: code = NotFound desc = could not find container \"997c8555c0db6c0beef10df5e2d0e7bad12b9a1db3bfcdc6f0cee42652f5beab\": container with ID starting with 997c8555c0db6c0beef10df5e2d0e7bad12b9a1db3bfcdc6f0cee42652f5beab not found: ID does not exist"
Sep 16 12:31:40.432696 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:40.432671    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"]
Sep 16 12:31:40.436774 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:40.436752    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7a4d5c-5668b7c8f7-t7849"]
Sep 16 12:31:41.213688 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:41.213647    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a048b31-681c-4da8-bdfd-7fe0d837fd67" path="/var/lib/kubelet/pods/4a048b31-681c-4da8-bdfd-7fe0d837fd67/volumes"
Sep 16 12:31:42.033988 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.033896    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm" podUID="5a9bb324-3182-43ff-b96b-03d578c69153" containerName="main" containerID="cri-o://9670624a9a2021cae55ec54fa13b3522b4dcd3bac5232d76e7537965c7456991" gracePeriod=45
Sep 16 12:31:42.048600 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.048552    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s" podUID="d1a3df43-e996-42d0-93e2-d13b60da875c" containerName="main" containerID="cri-o://ffef68f480536e8dd79ccfef41100f8329fab94e6959446340a6123ea93febca" gracePeriod=45
Sep 16 12:31:42.060177 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.060141    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw" podUID="e20e88fc-24e6-4ba3-b75b-e90da065f515" containerName="main" containerID="cri-o://ee01c8358dfad792afc40a955d6212097649003ec5bdc7c574a83d123d7865fa" gracePeriod=45
Sep 16 12:31:42.084475 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.084440    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"]
Sep 16 12:31:42.084700 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.084677    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp" podUID="a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59" containerName="maas-api" containerID="cri-o://f04ffa886b81e04fc955c4342147ff54d7439571933d3ce6a814a8d5bcf4add2" gracePeriod=30
Sep 16 12:31:42.387110 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.387075    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:31:42.404487 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.404388    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:31:42.415047 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.414400    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:31:42.417927 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.417901    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:31:42.425322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.425291    2582 generic.go:356] "Generic (PLEG): container finished" podID="d1a3df43-e996-42d0-93e2-d13b60da875c" containerID="ffef68f480536e8dd79ccfef41100f8329fab94e6959446340a6123ea93febca" exitCode=0
Sep 16 12:31:42.425481 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.425334    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"
Sep 16 12:31:42.425481 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.425369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s" event={"ID":"d1a3df43-e996-42d0-93e2-d13b60da875c","Type":"ContainerDied","Data":"ffef68f480536e8dd79ccfef41100f8329fab94e6959446340a6123ea93febca"}
Sep 16 12:31:42.425481 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.425421    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s" event={"ID":"d1a3df43-e996-42d0-93e2-d13b60da875c","Type":"ContainerDied","Data":"ebd9f089dd230a9668bd4d693ccc42a34954a6847c905bdb82c69f51d30d1e1f"}
Sep 16 12:31:42.425481 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.425452    2582 scope.go:117] "RemoveContainer" containerID="ffef68f480536e8dd79ccfef41100f8329fab94e6959446340a6123ea93febca"
Sep 16 12:31:42.427031 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.427006    2582 generic.go:356] "Generic (PLEG): container finished" podID="e20e88fc-24e6-4ba3-b75b-e90da065f515" containerID="ee01c8358dfad792afc40a955d6212097649003ec5bdc7c574a83d123d7865fa" exitCode=0
Sep 16 12:31:42.427192 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.427045    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"
Sep 16 12:31:42.427192 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.427113    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw" event={"ID":"e20e88fc-24e6-4ba3-b75b-e90da065f515","Type":"ContainerDied","Data":"ee01c8358dfad792afc40a955d6212097649003ec5bdc7c574a83d123d7865fa"}
Sep 16 12:31:42.427192 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.427142    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw" event={"ID":"e20e88fc-24e6-4ba3-b75b-e90da065f515","Type":"ContainerDied","Data":"ae249147558fed85dfe02b22bc1730b175f6ba544537c2c21706ba6347cf2510"}
Sep 16 12:31:42.430139 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.430110    2582 generic.go:356] "Generic (PLEG): container finished" podID="a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59" containerID="f04ffa886b81e04fc955c4342147ff54d7439571933d3ce6a814a8d5bcf4add2" exitCode=0
Sep 16 12:31:42.430250 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.430218    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"
Sep 16 12:31:42.430341 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.430322    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp" event={"ID":"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59","Type":"ContainerDied","Data":"f04ffa886b81e04fc955c4342147ff54d7439571933d3ce6a814a8d5bcf4add2"}
Sep 16 12:31:42.430405 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.430367    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp" event={"ID":"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59","Type":"ContainerDied","Data":"8e411e05dfcfe384c8595a6b277bdb81307878a523a3439b8579c7a7a382e989"}
Sep 16 12:31:42.432506 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.432478    2582 generic.go:356] "Generic (PLEG): container finished" podID="5a9bb324-3182-43ff-b96b-03d578c69153" containerID="9670624a9a2021cae55ec54fa13b3522b4dcd3bac5232d76e7537965c7456991" exitCode=0
Sep 16 12:31:42.433046 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.432553    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm" event={"ID":"5a9bb324-3182-43ff-b96b-03d578c69153","Type":"ContainerDied","Data":"9670624a9a2021cae55ec54fa13b3522b4dcd3bac5232d76e7537965c7456991"}
Sep 16 12:31:42.433046 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.432597    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm" event={"ID":"5a9bb324-3182-43ff-b96b-03d578c69153","Type":"ContainerDied","Data":"10e966cac32273c1c7153bfb7fc83bb1c586d2f77b921056005e1efe905351f2"}
Sep 16 12:31:42.433046 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.432652    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"
Sep 16 12:31:42.439324 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.439306    2582 scope.go:117] "RemoveContainer" containerID="ffef68f480536e8dd79ccfef41100f8329fab94e6959446340a6123ea93febca"
Sep 16 12:31:42.439640 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:31:42.439621    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffef68f480536e8dd79ccfef41100f8329fab94e6959446340a6123ea93febca\": container with ID starting with ffef68f480536e8dd79ccfef41100f8329fab94e6959446340a6123ea93febca not found: ID does not exist" containerID="ffef68f480536e8dd79ccfef41100f8329fab94e6959446340a6123ea93febca"
Sep 16 12:31:42.439711 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.439651    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffef68f480536e8dd79ccfef41100f8329fab94e6959446340a6123ea93febca"} err="failed to get container status \"ffef68f480536e8dd79ccfef41100f8329fab94e6959446340a6123ea93febca\": rpc error: code = NotFound desc = could not find container \"ffef68f480536e8dd79ccfef41100f8329fab94e6959446340a6123ea93febca\": container with ID starting with ffef68f480536e8dd79ccfef41100f8329fab94e6959446340a6123ea93febca not found: ID does not exist"
Sep 16 12:31:42.439711 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.439668    2582 scope.go:117] "RemoveContainer" containerID="ee01c8358dfad792afc40a955d6212097649003ec5bdc7c574a83d123d7865fa"
Sep 16 12:31:42.453460 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.453437    2582 scope.go:117] "RemoveContainer" containerID="ee01c8358dfad792afc40a955d6212097649003ec5bdc7c574a83d123d7865fa"
Sep 16 12:31:42.454243 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:31:42.454212    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee01c8358dfad792afc40a955d6212097649003ec5bdc7c574a83d123d7865fa\": container with ID starting with ee01c8358dfad792afc40a955d6212097649003ec5bdc7c574a83d123d7865fa not found: ID does not exist" containerID="ee01c8358dfad792afc40a955d6212097649003ec5bdc7c574a83d123d7865fa"
Sep 16 12:31:42.454363 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.454255    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee01c8358dfad792afc40a955d6212097649003ec5bdc7c574a83d123d7865fa"} err="failed to get container status \"ee01c8358dfad792afc40a955d6212097649003ec5bdc7c574a83d123d7865fa\": rpc error: code = NotFound desc = could not find container \"ee01c8358dfad792afc40a955d6212097649003ec5bdc7c574a83d123d7865fa\": container with ID starting with ee01c8358dfad792afc40a955d6212097649003ec5bdc7c574a83d123d7865fa not found: ID does not exist"
Sep 16 12:31:42.454363 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.454277    2582 scope.go:117] "RemoveContainer" containerID="f04ffa886b81e04fc955c4342147ff54d7439571933d3ce6a814a8d5bcf4add2"
Sep 16 12:31:42.465232 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.465209    2582 scope.go:117] "RemoveContainer" containerID="f04ffa886b81e04fc955c4342147ff54d7439571933d3ce6a814a8d5bcf4add2"
Sep 16 12:31:42.465611 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:31:42.465579    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f04ffa886b81e04fc955c4342147ff54d7439571933d3ce6a814a8d5bcf4add2\": container with ID starting with f04ffa886b81e04fc955c4342147ff54d7439571933d3ce6a814a8d5bcf4add2 not found: ID does not exist" containerID="f04ffa886b81e04fc955c4342147ff54d7439571933d3ce6a814a8d5bcf4add2"
Sep 16 12:31:42.465695 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.465622    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f04ffa886b81e04fc955c4342147ff54d7439571933d3ce6a814a8d5bcf4add2"} err="failed to get container status \"f04ffa886b81e04fc955c4342147ff54d7439571933d3ce6a814a8d5bcf4add2\": rpc error: code = NotFound desc = could not find container \"f04ffa886b81e04fc955c4342147ff54d7439571933d3ce6a814a8d5bcf4add2\": container with ID starting with f04ffa886b81e04fc955c4342147ff54d7439571933d3ce6a814a8d5bcf4add2 not found: ID does not exist"
Sep 16 12:31:42.465695 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.465644    2582 scope.go:117] "RemoveContainer" containerID="9670624a9a2021cae55ec54fa13b3522b4dcd3bac5232d76e7537965c7456991"
Sep 16 12:31:42.478113 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478081    2582 scope.go:117] "RemoveContainer" containerID="9670624a9a2021cae55ec54fa13b3522b4dcd3bac5232d76e7537965c7456991"
Sep 16 12:31:42.478430 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478409    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-model-cache\") pod \"d1a3df43-e996-42d0-93e2-d13b60da875c\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") "
Sep 16 12:31:42.478520 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:31:42.478443    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9670624a9a2021cae55ec54fa13b3522b4dcd3bac5232d76e7537965c7456991\": container with ID starting with 9670624a9a2021cae55ec54fa13b3522b4dcd3bac5232d76e7537965c7456991 not found: ID does not exist" containerID="9670624a9a2021cae55ec54fa13b3522b4dcd3bac5232d76e7537965c7456991"
Sep 16 12:31:42.478520 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478469    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9670624a9a2021cae55ec54fa13b3522b4dcd3bac5232d76e7537965c7456991"} err="failed to get container status \"9670624a9a2021cae55ec54fa13b3522b4dcd3bac5232d76e7537965c7456991\": rpc error: code = NotFound desc = could not find container \"9670624a9a2021cae55ec54fa13b3522b4dcd3bac5232d76e7537965c7456991\": container with ID starting with 9670624a9a2021cae55ec54fa13b3522b4dcd3bac5232d76e7537965c7456991 not found: ID does not exist"
Sep 16 12:31:42.478520 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478450    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-home\") pod \"d1a3df43-e996-42d0-93e2-d13b60da875c\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") "
Sep 16 12:31:42.479056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478535    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-tmp-dir\") pod \"5a9bb324-3182-43ff-b96b-03d578c69153\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") "
Sep 16 12:31:42.479056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478593    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lt29\" (UniqueName: \"kubernetes.io/projected/d1a3df43-e996-42d0-93e2-d13b60da875c-kube-api-access-7lt29\") pod \"d1a3df43-e996-42d0-93e2-d13b60da875c\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") "
Sep 16 12:31:42.479056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478622    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-model-cache\") pod \"5a9bb324-3182-43ff-b96b-03d578c69153\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") "
Sep 16 12:31:42.479056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478657    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26sfs\" (UniqueName: \"kubernetes.io/projected/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-kube-api-access-26sfs\") pod \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\" (UID: \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\") "
Sep 16 12:31:42.479056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478668    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-home" (OuterVolumeSpecName: "home") pod "d1a3df43-e996-42d0-93e2-d13b60da875c" (UID: "d1a3df43-e996-42d0-93e2-d13b60da875c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:42.479056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478685    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1a3df43-e996-42d0-93e2-d13b60da875c" (UID: "d1a3df43-e996-42d0-93e2-d13b60da875c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:42.479056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478697    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e20e88fc-24e6-4ba3-b75b-e90da065f515-tls-certs\") pod \"e20e88fc-24e6-4ba3-b75b-e90da065f515\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") "
Sep 16 12:31:42.479056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478729    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-maas-api-tls\") pod \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\" (UID: \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\") "
Sep 16 12:31:42.479056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478799    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-model-cache\") pod \"e20e88fc-24e6-4ba3-b75b-e90da065f515\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") "
Sep 16 12:31:42.479056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478825    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-home\") pod \"5a9bb324-3182-43ff-b96b-03d578c69153\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") "
Sep 16 12:31:42.479056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478860    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d662x\" (UniqueName: \"kubernetes.io/projected/e20e88fc-24e6-4ba3-b75b-e90da065f515-kube-api-access-d662x\") pod \"e20e88fc-24e6-4ba3-b75b-e90da065f515\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") "
Sep 16 12:31:42.479056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478901    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9bb324-3182-43ff-b96b-03d578c69153-tls-certs\") pod \"5a9bb324-3182-43ff-b96b-03d578c69153\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") "
Sep 16 12:31:42.479056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478920    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a9bb324-3182-43ff-b96b-03d578c69153" (UID: "5a9bb324-3182-43ff-b96b-03d578c69153"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:42.479056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478934    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-metrics-certs\") pod \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\" (UID: \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\") "
Sep 16 12:31:42.479056 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.478965    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-openshift-service-ca\") pod \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\" (UID: \"a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59\") "
Sep 16 12:31:42.480090 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.479109    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-model-cache" (OuterVolumeSpecName: "model-cache") pod "e20e88fc-24e6-4ba3-b75b-e90da065f515" (UID: "e20e88fc-24e6-4ba3-b75b-e90da065f515"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:42.480090 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.479326    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59" (UID: "a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:31:42.480090 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.479451    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a9bb324-3182-43ff-b96b-03d578c69153" (UID: "5a9bb324-3182-43ff-b96b-03d578c69153"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:42.480090 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.479772    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-home" (OuterVolumeSpecName: "home") pod "5a9bb324-3182-43ff-b96b-03d578c69153" (UID: "5a9bb324-3182-43ff-b96b-03d578c69153"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:42.480625 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.480604    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-tmp-dir\") pod \"e20e88fc-24e6-4ba3-b75b-e90da065f515\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") "
Sep 16 12:31:42.480722 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.480651    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-dshm\") pod \"d1a3df43-e996-42d0-93e2-d13b60da875c\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") "
Sep 16 12:31:42.480722 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.480690    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-dshm\") pod \"e20e88fc-24e6-4ba3-b75b-e90da065f515\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") "
Sep 16 12:31:42.480722 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.480714    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-dshm\") pod \"5a9bb324-3182-43ff-b96b-03d578c69153\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") "
Sep 16 12:31:42.481343 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.480807    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhgnh\" (UniqueName: \"kubernetes.io/projected/5a9bb324-3182-43ff-b96b-03d578c69153-kube-api-access-nhgnh\") pod \"5a9bb324-3182-43ff-b96b-03d578c69153\" (UID: \"5a9bb324-3182-43ff-b96b-03d578c69153\") "
Sep 16 12:31:42.481343 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.480848    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a3df43-e996-42d0-93e2-d13b60da875c-tls-certs\") pod \"d1a3df43-e996-42d0-93e2-d13b60da875c\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") "
Sep 16 12:31:42.481343 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.480876    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-tmp-dir\") pod \"d1a3df43-e996-42d0-93e2-d13b60da875c\" (UID: \"d1a3df43-e996-42d0-93e2-d13b60da875c\") "
Sep 16 12:31:42.481343 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.480904    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-home\") pod \"e20e88fc-24e6-4ba3-b75b-e90da065f515\" (UID: \"e20e88fc-24e6-4ba3-b75b-e90da065f515\") "
Sep 16 12:31:42.481343 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.481291    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.481343 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.481307    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.481343 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.481320    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.481343 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.481332    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.481343 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.481347    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.481893 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.481360    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.481893 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.481374    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.482108 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.482088    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-home" (OuterVolumeSpecName: "home") pod "e20e88fc-24e6-4ba3-b75b-e90da065f515" (UID: "e20e88fc-24e6-4ba3-b75b-e90da065f515"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:42.482684 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.482543    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20e88fc-24e6-4ba3-b75b-e90da065f515-kube-api-access-d662x" (OuterVolumeSpecName: "kube-api-access-d662x") pod "e20e88fc-24e6-4ba3-b75b-e90da065f515" (UID: "e20e88fc-24e6-4ba3-b75b-e90da065f515"). InnerVolumeSpecName "kube-api-access-d662x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:31:42.482684 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.482626    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a3df43-e996-42d0-93e2-d13b60da875c-kube-api-access-7lt29" (OuterVolumeSpecName: "kube-api-access-7lt29") pod "d1a3df43-e996-42d0-93e2-d13b60da875c" (UID: "d1a3df43-e996-42d0-93e2-d13b60da875c"). InnerVolumeSpecName "kube-api-access-7lt29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:31:42.482865 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.482719    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d1a3df43-e996-42d0-93e2-d13b60da875c" (UID: "d1a3df43-e996-42d0-93e2-d13b60da875c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:42.482865 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.482777    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e20e88fc-24e6-4ba3-b75b-e90da065f515" (UID: "e20e88fc-24e6-4ba3-b75b-e90da065f515"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:42.484598 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.484565    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-dshm" (OuterVolumeSpecName: "dshm") pod "5a9bb324-3182-43ff-b96b-03d578c69153" (UID: "5a9bb324-3182-43ff-b96b-03d578c69153"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:42.484840 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.484681    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9bb324-3182-43ff-b96b-03d578c69153-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a9bb324-3182-43ff-b96b-03d578c69153" (UID: "5a9bb324-3182-43ff-b96b-03d578c69153"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:31:42.485376 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.485328    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e20e88fc-24e6-4ba3-b75b-e90da065f515-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e20e88fc-24e6-4ba3-b75b-e90da065f515" (UID: "e20e88fc-24e6-4ba3-b75b-e90da065f515"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:31:42.485463 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.485384    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-kube-api-access-26sfs" (OuterVolumeSpecName: "kube-api-access-26sfs") pod "a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59" (UID: "a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59"). InnerVolumeSpecName "kube-api-access-26sfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:31:42.485463 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.485433    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-dshm" (OuterVolumeSpecName: "dshm") pod "e20e88fc-24e6-4ba3-b75b-e90da065f515" (UID: "e20e88fc-24e6-4ba3-b75b-e90da065f515"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:42.485611 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.485590    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59" (UID: "a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:31:42.485759 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.485723    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a3df43-e996-42d0-93e2-d13b60da875c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1a3df43-e996-42d0-93e2-d13b60da875c" (UID: "d1a3df43-e996-42d0-93e2-d13b60da875c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:31:42.485933 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.485912    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59" (UID: "a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:31:42.486758 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.486700    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9bb324-3182-43ff-b96b-03d578c69153-kube-api-access-nhgnh" (OuterVolumeSpecName: "kube-api-access-nhgnh") pod "5a9bb324-3182-43ff-b96b-03d578c69153" (UID: "5a9bb324-3182-43ff-b96b-03d578c69153"). InnerVolumeSpecName "kube-api-access-nhgnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:31:42.487092 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.487074    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-dshm" (OuterVolumeSpecName: "dshm") pod "d1a3df43-e996-42d0-93e2-d13b60da875c" (UID: "d1a3df43-e996-42d0-93e2-d13b60da875c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:42.582809 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.582779    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhgnh\" (UniqueName: \"kubernetes.io/projected/5a9bb324-3182-43ff-b96b-03d578c69153-kube-api-access-nhgnh\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.582809 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.582808    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a3df43-e996-42d0-93e2-d13b60da875c-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.583034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.582818    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.583034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.582827    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.583034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.582836    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lt29\" (UniqueName: \"kubernetes.io/projected/d1a3df43-e996-42d0-93e2-d13b60da875c-kube-api-access-7lt29\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.583034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.582845    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26sfs\" (UniqueName: \"kubernetes.io/projected/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-kube-api-access-26sfs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.583034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.582853    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e20e88fc-24e6-4ba3-b75b-e90da065f515-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.583034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.582861    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.583034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.582869    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d662x\" (UniqueName: \"kubernetes.io/projected/e20e88fc-24e6-4ba3-b75b-e90da065f515-kube-api-access-d662x\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.583034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.582876    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9bb324-3182-43ff-b96b-03d578c69153-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.583034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.582884    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.583034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.582891    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.583034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.582898    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a3df43-e996-42d0-93e2-d13b60da875c-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.583034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.582906    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e20e88fc-24e6-4ba3-b75b-e90da065f515-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.583034 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.582913    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9bb324-3182-43ff-b96b-03d578c69153-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.765924 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.765896    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"]
Sep 16 12:31:42.771362 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.771337    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/premium-w4-7a4d5c-kserve-9f7c78cd4-8t97s"]
Sep 16 12:31:42.782158 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.782134    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"]
Sep 16 12:31:42.786687 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.786659    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-835ec8-69f7898945-7q6kp"]
Sep 16 12:31:42.798153 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.798122    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"]
Sep 16 12:31:42.801917 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.801888    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/simulator-w4-7a4d5c-kserve-5cf8dd84c7-szdpw"]
Sep 16 12:31:42.821890 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.821852    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"]
Sep 16 12:31:42.825351 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:42.825324    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7a4d5c/distinct-w4-7a4d5c-kserve-568df9bf9-qv6vm"]
Sep 16 12:31:43.212897 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:43.212858    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a9bb324-3182-43ff-b96b-03d578c69153" path="/var/lib/kubelet/pods/5a9bb324-3182-43ff-b96b-03d578c69153/volumes"
Sep 16 12:31:43.213295 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:43.213280    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59" path="/var/lib/kubelet/pods/a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59/volumes"
Sep 16 12:31:43.213668 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:43.213653    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1a3df43-e996-42d0-93e2-d13b60da875c" path="/var/lib/kubelet/pods/d1a3df43-e996-42d0-93e2-d13b60da875c/volumes"
Sep 16 12:31:43.214051 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:31:43.214036    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e20e88fc-24e6-4ba3-b75b-e90da065f515" path="/var/lib/kubelet/pods/e20e88fc-24e6-4ba3-b75b-e90da065f515/volumes"
Sep 16 12:32:05.376283 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:05.376197    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"]
Sep 16 12:32:05.376759 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:05.376465    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk" podUID="89b63944-f9c6-4f4b-b562-e2ab196c34ac" containerName="maas-api" containerID="cri-o://f6b3409e780d0d81184be5d170e77c90e069ca7e1c92dc5216ee72e6db2f5025" gracePeriod=30
Sep 16 12:32:05.576841 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:05.576801    2582 generic.go:356] "Generic (PLEG): container finished" podID="89b63944-f9c6-4f4b-b562-e2ab196c34ac" containerID="f6b3409e780d0d81184be5d170e77c90e069ca7e1c92dc5216ee72e6db2f5025" exitCode=0
Sep 16 12:32:05.577047 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:05.576851    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk" event={"ID":"89b63944-f9c6-4f4b-b562-e2ab196c34ac","Type":"ContainerDied","Data":"f6b3409e780d0d81184be5d170e77c90e069ca7e1c92dc5216ee72e6db2f5025"}
Sep 16 12:32:05.627002 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:05.626928    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:32:05.705514 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:05.705475    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b63944-f9c6-4f4b-b562-e2ab196c34ac-maas-api-tls\") pod \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\" (UID: \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\") "
Sep 16 12:32:05.705514 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:05.705517    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b63944-f9c6-4f4b-b562-e2ab196c34ac-metrics-certs\") pod \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\" (UID: \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\") "
Sep 16 12:32:05.705778 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:05.705556    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b63944-f9c6-4f4b-b562-e2ab196c34ac-openshift-service-ca\") pod \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\" (UID: \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\") "
Sep 16 12:32:05.705778 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:05.705603    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkrfw\" (UniqueName: \"kubernetes.io/projected/89b63944-f9c6-4f4b-b562-e2ab196c34ac-kube-api-access-zkrfw\") pod \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\" (UID: \"89b63944-f9c6-4f4b-b562-e2ab196c34ac\") "
Sep 16 12:32:05.705968 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:05.705941    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b63944-f9c6-4f4b-b562-e2ab196c34ac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "89b63944-f9c6-4f4b-b562-e2ab196c34ac" (UID: "89b63944-f9c6-4f4b-b562-e2ab196c34ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:32:05.707964 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:05.707937    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b63944-f9c6-4f4b-b562-e2ab196c34ac-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "89b63944-f9c6-4f4b-b562-e2ab196c34ac" (UID: "89b63944-f9c6-4f4b-b562-e2ab196c34ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:32:05.708084 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:05.707955    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b63944-f9c6-4f4b-b562-e2ab196c34ac-kube-api-access-zkrfw" (OuterVolumeSpecName: "kube-api-access-zkrfw") pod "89b63944-f9c6-4f4b-b562-e2ab196c34ac" (UID: "89b63944-f9c6-4f4b-b562-e2ab196c34ac"). InnerVolumeSpecName "kube-api-access-zkrfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:05.708084 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:05.708031    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b63944-f9c6-4f4b-b562-e2ab196c34ac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "89b63944-f9c6-4f4b-b562-e2ab196c34ac" (UID: "89b63944-f9c6-4f4b-b562-e2ab196c34ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:32:05.806945 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:05.806902    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b63944-f9c6-4f4b-b562-e2ab196c34ac-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:05.806945 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:05.806937    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b63944-f9c6-4f4b-b562-e2ab196c34ac-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:05.806945 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:05.806947    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b63944-f9c6-4f4b-b562-e2ab196c34ac-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:05.806945 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:05.806956    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkrfw\" (UniqueName: \"kubernetes.io/projected/89b63944-f9c6-4f4b-b562-e2ab196c34ac-kube-api-access-zkrfw\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:06.584211 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:06.584187    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"
Sep 16 12:32:06.584655 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:06.584236    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk" event={"ID":"89b63944-f9c6-4f4b-b562-e2ab196c34ac","Type":"ContainerDied","Data":"bc078bb5f6b367dcd2a9bed20e89489af26067884c1585e92eeffed12fb23ba9"}
Sep 16 12:32:06.584655 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:06.584304    2582 scope.go:117] "RemoveContainer" containerID="f6b3409e780d0d81184be5d170e77c90e069ca7e1c92dc5216ee72e6db2f5025"
Sep 16 12:32:06.606084 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:06.606048    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"]
Sep 16 12:32:06.610345 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:06.610314    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5c41db-66bfdc64c-l2slk"]
Sep 16 12:32:07.212458 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:07.212421    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89b63944-f9c6-4f4b-b562-e2ab196c34ac" path="/var/lib/kubelet/pods/89b63944-f9c6-4f4b-b562-e2ab196c34ac/volumes"
Sep 16 12:32:12.339322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.339283    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7567-1f1fdf"]
Sep 16 12:32:12.339902 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.339879    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a9bb324-3182-43ff-b96b-03d578c69153" containerName="main"
Sep 16 12:32:12.339964 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.339906    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a9bb324-3182-43ff-b96b-03d578c69153" containerName="main"
Sep 16 12:32:12.339964 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.339934    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1a3df43-e996-42d0-93e2-d13b60da875c" containerName="main"
Sep 16 12:32:12.339964 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.339943    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1a3df43-e996-42d0-93e2-d13b60da875c" containerName="main"
Sep 16 12:32:12.339964 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.339955    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59" containerName="maas-api"
Sep 16 12:32:12.339964 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.339964    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59" containerName="maas-api"
Sep 16 12:32:12.340120 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.339976    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a048b31-681c-4da8-bdfd-7fe0d837fd67" containerName="maas-api"
Sep 16 12:32:12.340120 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.339984    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a048b31-681c-4da8-bdfd-7fe0d837fd67" containerName="maas-api"
Sep 16 12:32:12.340120 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.339996    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e20e88fc-24e6-4ba3-b75b-e90da065f515" containerName="main"
Sep 16 12:32:12.340120 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.340005    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e20e88fc-24e6-4ba3-b75b-e90da065f515" containerName="main"
Sep 16 12:32:12.340120 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.340030    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b63944-f9c6-4f4b-b562-e2ab196c34ac" containerName="maas-api"
Sep 16 12:32:12.340120 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.340039    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b63944-f9c6-4f4b-b562-e2ab196c34ac" containerName="maas-api"
Sep 16 12:32:12.340307 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.340124    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1a3df43-e996-42d0-93e2-d13b60da875c" containerName="main"
Sep 16 12:32:12.340307 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.340138    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e20e88fc-24e6-4ba3-b75b-e90da065f515" containerName="main"
Sep 16 12:32:12.340307 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.340146    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a048b31-681c-4da8-bdfd-7fe0d837fd67" containerName="maas-api"
Sep 16 12:32:12.340307 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.340156    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a9bb324-3182-43ff-b96b-03d578c69153" containerName="main"
Sep 16 12:32:12.340307 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.340167    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="89b63944-f9c6-4f4b-b562-e2ab196c34ac" containerName="maas-api"
Sep 16 12:32:12.340307 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.340175    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e76409-a2c4-4ece-bf9f-0aa0c8d4ca59" containerName="maas-api"
Sep 16 12:32:12.344907 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.344884    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-1f1fdf"
Sep 16 12:32:12.348518 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.348497    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h8vfr\""
Sep 16 12:32:12.356561 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.356535    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-1f1fdf"]
Sep 16 12:32:12.368896 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.368865    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cssr\" (UniqueName: \"kubernetes.io/projected/265104ae-4954-451b-ad38-934f9e35f843-kube-api-access-4cssr\") pod \"test-curl-7567-1f1fdf\" (UID: \"265104ae-4954-451b-ad38-934f9e35f843\") " pod="opendatahub/test-curl-7567-1f1fdf"
Sep 16 12:32:12.470004 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.469973    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cssr\" (UniqueName: \"kubernetes.io/projected/265104ae-4954-451b-ad38-934f9e35f843-kube-api-access-4cssr\") pod \"test-curl-7567-1f1fdf\" (UID: \"265104ae-4954-451b-ad38-934f9e35f843\") " pod="opendatahub/test-curl-7567-1f1fdf"
Sep 16 12:32:12.479860 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.479828    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cssr\" (UniqueName: \"kubernetes.io/projected/265104ae-4954-451b-ad38-934f9e35f843-kube-api-access-4cssr\") pod \"test-curl-7567-1f1fdf\" (UID: \"265104ae-4954-451b-ad38-934f9e35f843\") " pod="opendatahub/test-curl-7567-1f1fdf"
Sep 16 12:32:12.655662 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.655555    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-1f1fdf"
Sep 16 12:32:12.925267 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:12.925234    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-1f1fdf"]
Sep 16 12:32:12.927288 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:32:12.927258    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod265104ae_4954_451b_ad38_934f9e35f843.slice/crio-976ae069e4d7a8d314838d485a42c362503e4f203d0add82a82cb123d787fc31 WatchSource:0}: Error finding container 976ae069e4d7a8d314838d485a42c362503e4f203d0add82a82cb123d787fc31: Status 404 returned error can't find the container with id 976ae069e4d7a8d314838d485a42c362503e4f203d0add82a82cb123d787fc31
Sep 16 12:32:13.632790 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:13.632750    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-1f1fdf" event={"ID":"265104ae-4954-451b-ad38-934f9e35f843","Type":"ContainerStarted","Data":"9b09b786193327034c94a25cb7eeda0de0ce09a758fbcae0595ec2b3aa0a14db"}
Sep 16 12:32:13.632790 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:13.632794    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-1f1fdf" event={"ID":"265104ae-4954-451b-ad38-934f9e35f843","Type":"ContainerStarted","Data":"976ae069e4d7a8d314838d485a42c362503e4f203d0add82a82cb123d787fc31"}
Sep 16 12:32:13.651014 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:13.650962    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7567-1f1fdf" podStartSLOduration=1.650948104 podStartE2EDuration="1.650948104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:32:13.647937995 +0000 UTC m=+1892.898475555" watchObservedRunningTime="2026-09-16 12:32:13.650948104 +0000 UTC m=+1892.901485652"
Sep 16 12:32:14.282809 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:14.282774    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7567-1f1fdf"]
Sep 16 12:32:14.285636 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:14.285607    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7567-1f1fdf"]
Sep 16 12:32:14.638927 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:32:14.638890    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7567-1f1fdf" container="test-curl-7567-1f1fdf"
Sep 16 12:32:15.644532 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:15.644465    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7567-1f1fdf" podUID="265104ae-4954-451b-ad38-934f9e35f843" containerName="test-curl-7567-1f1fdf" containerID="cri-o://9b09b786193327034c94a25cb7eeda0de0ce09a758fbcae0595ec2b3aa0a14db" gracePeriod=2
Sep 16 12:32:15.707680 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:15.707646    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"]
Sep 16 12:32:16.710828 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:16.710721    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"]
Sep 16 12:32:17.839809 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:17.839773    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-1f1fdf"
Sep 16 12:32:17.921947 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:17.921842    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cssr\" (UniqueName: \"kubernetes.io/projected/265104ae-4954-451b-ad38-934f9e35f843-kube-api-access-4cssr\") pod \"265104ae-4954-451b-ad38-934f9e35f843\" (UID: \"265104ae-4954-451b-ad38-934f9e35f843\") "
Sep 16 12:32:17.924131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:17.924097    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265104ae-4954-451b-ad38-934f9e35f843-kube-api-access-4cssr" (OuterVolumeSpecName: "kube-api-access-4cssr") pod "265104ae-4954-451b-ad38-934f9e35f843" (UID: "265104ae-4954-451b-ad38-934f9e35f843"). InnerVolumeSpecName "kube-api-access-4cssr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:18.023639 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:18.023605    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cssr\" (UniqueName: \"kubernetes.io/projected/265104ae-4954-451b-ad38-934f9e35f843-kube-api-access-4cssr\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:18.666848 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:18.666811    2582 generic.go:356] "Generic (PLEG): container finished" podID="265104ae-4954-451b-ad38-934f9e35f843" containerID="9b09b786193327034c94a25cb7eeda0de0ce09a758fbcae0595ec2b3aa0a14db" exitCode=137
Sep 16 12:32:18.667036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:18.666855    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-1f1fdf"
Sep 16 12:32:18.667036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:18.666879    2582 scope.go:117] "RemoveContainer" containerID="9b09b786193327034c94a25cb7eeda0de0ce09a758fbcae0595ec2b3aa0a14db"
Sep 16 12:32:18.676281 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:18.676261    2582 scope.go:117] "RemoveContainer" containerID="9b09b786193327034c94a25cb7eeda0de0ce09a758fbcae0595ec2b3aa0a14db"
Sep 16 12:32:18.676579 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:32:18.676561    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b09b786193327034c94a25cb7eeda0de0ce09a758fbcae0595ec2b3aa0a14db\": container with ID starting with 9b09b786193327034c94a25cb7eeda0de0ce09a758fbcae0595ec2b3aa0a14db not found: ID does not exist" containerID="9b09b786193327034c94a25cb7eeda0de0ce09a758fbcae0595ec2b3aa0a14db"
Sep 16 12:32:18.676627 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:18.676597    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b09b786193327034c94a25cb7eeda0de0ce09a758fbcae0595ec2b3aa0a14db"} err="failed to get container status \"9b09b786193327034c94a25cb7eeda0de0ce09a758fbcae0595ec2b3aa0a14db\": rpc error: code = NotFound desc = could not find container \"9b09b786193327034c94a25cb7eeda0de0ce09a758fbcae0595ec2b3aa0a14db\": container with ID starting with 9b09b786193327034c94a25cb7eeda0de0ce09a758fbcae0595ec2b3aa0a14db not found: ID does not exist"
Sep 16 12:32:18.677058 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:32:18.677033    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7567-1f1fdf\" is forbidden: User \"system:node:ip-10-0-140-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-71.ec2.internal' and this object" podUID="265104ae-4954-451b-ad38-934f9e35f843" pod="opendatahub/test-curl-7567-1f1fdf"
Sep 16 12:32:19.212227 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:19.212179    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="265104ae-4954-451b-ad38-934f9e35f843" path="/var/lib/kubelet/pods/265104ae-4954-451b-ad38-934f9e35f843/volumes"
Sep 16 12:32:23.071957 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.071922    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"]
Sep 16 12:32:23.072441 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.072185    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl" podUID="06773e03-fd60-43ae-aa35-8dd3b5f6a665" containerName="maas-api" containerID="cri-o://5970a6a4e4269604862882c969a961140535b9a6fa3dfe16b0fe067a958e8589" gracePeriod=30
Sep 16 12:32:23.322130 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.322063    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:32:23.373390 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.373359    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06773e03-fd60-43ae-aa35-8dd3b5f6a665-maas-api-tls\") pod \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\" (UID: \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\") "
Sep 16 12:32:23.373536 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.373465    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06773e03-fd60-43ae-aa35-8dd3b5f6a665-metrics-certs\") pod \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\" (UID: \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\") "
Sep 16 12:32:23.373536 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.373486    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06773e03-fd60-43ae-aa35-8dd3b5f6a665-openshift-service-ca\") pod \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\" (UID: \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\") "
Sep 16 12:32:23.373536 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.373526    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw46f\" (UniqueName: \"kubernetes.io/projected/06773e03-fd60-43ae-aa35-8dd3b5f6a665-kube-api-access-lw46f\") pod \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\" (UID: \"06773e03-fd60-43ae-aa35-8dd3b5f6a665\") "
Sep 16 12:32:23.373928 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.373895    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06773e03-fd60-43ae-aa35-8dd3b5f6a665-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06773e03-fd60-43ae-aa35-8dd3b5f6a665" (UID: "06773e03-fd60-43ae-aa35-8dd3b5f6a665"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:32:23.375818 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.375787    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06773e03-fd60-43ae-aa35-8dd3b5f6a665-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "06773e03-fd60-43ae-aa35-8dd3b5f6a665" (UID: "06773e03-fd60-43ae-aa35-8dd3b5f6a665"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:32:23.375935 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.375821    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06773e03-fd60-43ae-aa35-8dd3b5f6a665-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06773e03-fd60-43ae-aa35-8dd3b5f6a665" (UID: "06773e03-fd60-43ae-aa35-8dd3b5f6a665"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:32:23.375935 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.375894    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06773e03-fd60-43ae-aa35-8dd3b5f6a665-kube-api-access-lw46f" (OuterVolumeSpecName: "kube-api-access-lw46f") pod "06773e03-fd60-43ae-aa35-8dd3b5f6a665" (UID: "06773e03-fd60-43ae-aa35-8dd3b5f6a665"). InnerVolumeSpecName "kube-api-access-lw46f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:23.475119 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.475089    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06773e03-fd60-43ae-aa35-8dd3b5f6a665-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:23.475119 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.475114    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06773e03-fd60-43ae-aa35-8dd3b5f6a665-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:23.475119 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.475123    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06773e03-fd60-43ae-aa35-8dd3b5f6a665-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:23.475435 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.475132    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw46f\" (UniqueName: \"kubernetes.io/projected/06773e03-fd60-43ae-aa35-8dd3b5f6a665-kube-api-access-lw46f\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:23.696933 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.696843    2582 generic.go:356] "Generic (PLEG): container finished" podID="06773e03-fd60-43ae-aa35-8dd3b5f6a665" containerID="5970a6a4e4269604862882c969a961140535b9a6fa3dfe16b0fe067a958e8589" exitCode=0
Sep 16 12:32:23.696933 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.696898    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"
Sep 16 12:32:23.696933 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.696922    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl" event={"ID":"06773e03-fd60-43ae-aa35-8dd3b5f6a665","Type":"ContainerDied","Data":"5970a6a4e4269604862882c969a961140535b9a6fa3dfe16b0fe067a958e8589"}
Sep 16 12:32:23.697177 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.696964    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl" event={"ID":"06773e03-fd60-43ae-aa35-8dd3b5f6a665","Type":"ContainerDied","Data":"0825b081360cd6965925e2c4d99346799975bc52a136004af2188fd9d39b5f48"}
Sep 16 12:32:23.697177 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.696985    2582 scope.go:117] "RemoveContainer" containerID="5970a6a4e4269604862882c969a961140535b9a6fa3dfe16b0fe067a958e8589"
Sep 16 12:32:23.707040 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.707020    2582 scope.go:117] "RemoveContainer" containerID="5970a6a4e4269604862882c969a961140535b9a6fa3dfe16b0fe067a958e8589"
Sep 16 12:32:23.707351 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:32:23.707317    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5970a6a4e4269604862882c969a961140535b9a6fa3dfe16b0fe067a958e8589\": container with ID starting with 5970a6a4e4269604862882c969a961140535b9a6fa3dfe16b0fe067a958e8589 not found: ID does not exist" containerID="5970a6a4e4269604862882c969a961140535b9a6fa3dfe16b0fe067a958e8589"
Sep 16 12:32:23.707403 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.707365    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5970a6a4e4269604862882c969a961140535b9a6fa3dfe16b0fe067a958e8589"} err="failed to get container status \"5970a6a4e4269604862882c969a961140535b9a6fa3dfe16b0fe067a958e8589\": rpc error: code = NotFound desc = could not find container \"5970a6a4e4269604862882c969a961140535b9a6fa3dfe16b0fe067a958e8589\": container with ID starting with 5970a6a4e4269604862882c969a961140535b9a6fa3dfe16b0fe067a958e8589 not found: ID does not exist"
Sep 16 12:32:23.730301 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.725137    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"]
Sep 16 12:32:23.734793 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:23.734764    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-65d8af-5fbc984bf8-68qxl"]
Sep 16 12:32:25.212048 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:25.212014    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06773e03-fd60-43ae-aa35-8dd3b5f6a665" path="/var/lib/kubelet/pods/06773e03-fd60-43ae-aa35-8dd3b5f6a665/volumes"
Sep 16 12:32:30.728215 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:30.728154    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z" podUID="1b5f731c-b509-4f45-a119-e7d4e520ac63" containerName="main" containerID="cri-o://6a3990abe042e3ceb6efb6193076089f4c82e38f6adff50408d03ab2e22576d3" gracePeriod=45
Sep 16 12:32:30.982091 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:30.982025    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:32:31.042104 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.042064    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-model-cache\") pod \"1b5f731c-b509-4f45-a119-e7d4e520ac63\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") "
Sep 16 12:32:31.042104 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.042104    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-home\") pod \"1b5f731c-b509-4f45-a119-e7d4e520ac63\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") "
Sep 16 12:32:31.042348 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.042123    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnwlk\" (UniqueName: \"kubernetes.io/projected/1b5f731c-b509-4f45-a119-e7d4e520ac63-kube-api-access-cnwlk\") pod \"1b5f731c-b509-4f45-a119-e7d4e520ac63\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") "
Sep 16 12:32:31.042348 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.042194    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-tmp-dir\") pod \"1b5f731c-b509-4f45-a119-e7d4e520ac63\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") "
Sep 16 12:32:31.042348 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.042238    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-dshm\") pod \"1b5f731c-b509-4f45-a119-e7d4e520ac63\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") "
Sep 16 12:32:31.042348 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.042278    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b5f731c-b509-4f45-a119-e7d4e520ac63-tls-certs\") pod \"1b5f731c-b509-4f45-a119-e7d4e520ac63\" (UID: \"1b5f731c-b509-4f45-a119-e7d4e520ac63\") "
Sep 16 12:32:31.042565 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.042393    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b5f731c-b509-4f45-a119-e7d4e520ac63" (UID: "1b5f731c-b509-4f45-a119-e7d4e520ac63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:31.042565 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.042404    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-home" (OuterVolumeSpecName: "home") pod "1b5f731c-b509-4f45-a119-e7d4e520ac63" (UID: "1b5f731c-b509-4f45-a119-e7d4e520ac63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:31.042565 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.042443    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b5f731c-b509-4f45-a119-e7d4e520ac63" (UID: "1b5f731c-b509-4f45-a119-e7d4e520ac63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:31.042722 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.042599    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:31.042722 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.042611    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:31.042722 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.042620    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:31.044631 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.044608    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-dshm" (OuterVolumeSpecName: "dshm") pod "1b5f731c-b509-4f45-a119-e7d4e520ac63" (UID: "1b5f731c-b509-4f45-a119-e7d4e520ac63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:31.044631 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.044619    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b5f731c-b509-4f45-a119-e7d4e520ac63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b5f731c-b509-4f45-a119-e7d4e520ac63" (UID: "1b5f731c-b509-4f45-a119-e7d4e520ac63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:32:31.044823 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.044663    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5f731c-b509-4f45-a119-e7d4e520ac63-kube-api-access-cnwlk" (OuterVolumeSpecName: "kube-api-access-cnwlk") pod "1b5f731c-b509-4f45-a119-e7d4e520ac63" (UID: "1b5f731c-b509-4f45-a119-e7d4e520ac63"). InnerVolumeSpecName "kube-api-access-cnwlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:31.143969 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.143915    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnwlk\" (UniqueName: \"kubernetes.io/projected/1b5f731c-b509-4f45-a119-e7d4e520ac63-kube-api-access-cnwlk\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:31.143969 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.143959    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b5f731c-b509-4f45-a119-e7d4e520ac63-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:31.143969 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.143970    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b5f731c-b509-4f45-a119-e7d4e520ac63-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:31.731334 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.731297    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8" podUID="0741457e-9313-48d6-a2ce-0f12dd916445" containerName="main" containerID="cri-o://ab0f9e103eda50e270e4d3c8d44e0da13f31804a4a2cbfbb87b1c4842570cc08" gracePeriod=45
Sep 16 12:32:31.744816 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.744771    2582 generic.go:356] "Generic (PLEG): container finished" podID="1b5f731c-b509-4f45-a119-e7d4e520ac63" containerID="6a3990abe042e3ceb6efb6193076089f4c82e38f6adff50408d03ab2e22576d3" exitCode=0
Sep 16 12:32:31.744927 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.744829    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z" event={"ID":"1b5f731c-b509-4f45-a119-e7d4e520ac63","Type":"ContainerDied","Data":"6a3990abe042e3ceb6efb6193076089f4c82e38f6adff50408d03ab2e22576d3"}
Sep 16 12:32:31.744927 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.744858    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"
Sep 16 12:32:31.744927 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.744880    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z" event={"ID":"1b5f731c-b509-4f45-a119-e7d4e520ac63","Type":"ContainerDied","Data":"95b5304a39471bab346b86f9e05d9412db7e52e1fca3e36bf523870a6217c131"}
Sep 16 12:32:31.744927 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.744905    2582 scope.go:117] "RemoveContainer" containerID="6a3990abe042e3ceb6efb6193076089f4c82e38f6adff50408d03ab2e22576d3"
Sep 16 12:32:31.809241 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.809202    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"]
Sep 16 12:32:31.812161 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.812133    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/auth-test-model-72eee5-kserve-6bdbb7d699-8vz5z"]
Sep 16 12:32:31.843235 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.843215    2582 scope.go:117] "RemoveContainer" containerID="6a3990abe042e3ceb6efb6193076089f4c82e38f6adff50408d03ab2e22576d3"
Sep 16 12:32:31.843641 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:32:31.843614    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a3990abe042e3ceb6efb6193076089f4c82e38f6adff50408d03ab2e22576d3\": container with ID starting with 6a3990abe042e3ceb6efb6193076089f4c82e38f6adff50408d03ab2e22576d3 not found: ID does not exist" containerID="6a3990abe042e3ceb6efb6193076089f4c82e38f6adff50408d03ab2e22576d3"
Sep 16 12:32:31.843846 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.843658    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a3990abe042e3ceb6efb6193076089f4c82e38f6adff50408d03ab2e22576d3"} err="failed to get container status \"6a3990abe042e3ceb6efb6193076089f4c82e38f6adff50408d03ab2e22576d3\": rpc error: code = NotFound desc = could not find container \"6a3990abe042e3ceb6efb6193076089f4c82e38f6adff50408d03ab2e22576d3\": container with ID starting with 6a3990abe042e3ceb6efb6193076089f4c82e38f6adff50408d03ab2e22576d3 not found: ID does not exist"
Sep 16 12:32:31.981540 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:31.981472    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:32:32.053176 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.053140    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d57gs\" (UniqueName: \"kubernetes.io/projected/0741457e-9313-48d6-a2ce-0f12dd916445-kube-api-access-d57gs\") pod \"0741457e-9313-48d6-a2ce-0f12dd916445\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") "
Sep 16 12:32:32.053378 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.053197    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-model-cache\") pod \"0741457e-9313-48d6-a2ce-0f12dd916445\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") "
Sep 16 12:32:32.053378 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.053264    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-tmp-dir\") pod \"0741457e-9313-48d6-a2ce-0f12dd916445\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") "
Sep 16 12:32:32.053378 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.053280    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-home\") pod \"0741457e-9313-48d6-a2ce-0f12dd916445\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") "
Sep 16 12:32:32.053378 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.053297    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0741457e-9313-48d6-a2ce-0f12dd916445-tls-certs\") pod \"0741457e-9313-48d6-a2ce-0f12dd916445\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") "
Sep 16 12:32:32.053378 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.053366    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-dshm\") pod \"0741457e-9313-48d6-a2ce-0f12dd916445\" (UID: \"0741457e-9313-48d6-a2ce-0f12dd916445\") "
Sep 16 12:32:32.053642 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.053485    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-model-cache" (OuterVolumeSpecName: "model-cache") pod "0741457e-9313-48d6-a2ce-0f12dd916445" (UID: "0741457e-9313-48d6-a2ce-0f12dd916445"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:32.053642 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.053504    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0741457e-9313-48d6-a2ce-0f12dd916445" (UID: "0741457e-9313-48d6-a2ce-0f12dd916445"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:32.053642 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.053517    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-home" (OuterVolumeSpecName: "home") pod "0741457e-9313-48d6-a2ce-0f12dd916445" (UID: "0741457e-9313-48d6-a2ce-0f12dd916445"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:32.053837 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.053671    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:32.053837 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.053688    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:32.053837 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.053698    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:32.055788 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.055713    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0741457e-9313-48d6-a2ce-0f12dd916445-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0741457e-9313-48d6-a2ce-0f12dd916445" (UID: "0741457e-9313-48d6-a2ce-0f12dd916445"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:32:32.055788 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.055712    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-dshm" (OuterVolumeSpecName: "dshm") pod "0741457e-9313-48d6-a2ce-0f12dd916445" (UID: "0741457e-9313-48d6-a2ce-0f12dd916445"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:32.055788 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.055726    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0741457e-9313-48d6-a2ce-0f12dd916445-kube-api-access-d57gs" (OuterVolumeSpecName: "kube-api-access-d57gs") pod "0741457e-9313-48d6-a2ce-0f12dd916445" (UID: "0741457e-9313-48d6-a2ce-0f12dd916445"). InnerVolumeSpecName "kube-api-access-d57gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:32.154806 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.154767    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0741457e-9313-48d6-a2ce-0f12dd916445-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:32.154806 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.154799    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d57gs\" (UniqueName: \"kubernetes.io/projected/0741457e-9313-48d6-a2ce-0f12dd916445-kube-api-access-d57gs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:32.154806 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.154810    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0741457e-9313-48d6-a2ce-0f12dd916445-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:32.752918 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.752885    2582 generic.go:356] "Generic (PLEG): container finished" podID="0741457e-9313-48d6-a2ce-0f12dd916445" containerID="ab0f9e103eda50e270e4d3c8d44e0da13f31804a4a2cbfbb87b1c4842570cc08" exitCode=0
Sep 16 12:32:32.752918 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.752923    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8" event={"ID":"0741457e-9313-48d6-a2ce-0f12dd916445","Type":"ContainerDied","Data":"ab0f9e103eda50e270e4d3c8d44e0da13f31804a4a2cbfbb87b1c4842570cc08"}
Sep 16 12:32:32.753432 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.752932    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"
Sep 16 12:32:32.753432 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.752955    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8" event={"ID":"0741457e-9313-48d6-a2ce-0f12dd916445","Type":"ContainerDied","Data":"a55528431cac922c6df24a61c0db9e710f46ac64ba2addb2a40d5fbeca282ff0"}
Sep 16 12:32:32.753432 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.752975    2582 scope.go:117] "RemoveContainer" containerID="ab0f9e103eda50e270e4d3c8d44e0da13f31804a4a2cbfbb87b1c4842570cc08"
Sep 16 12:32:32.763022 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.763001    2582 scope.go:117] "RemoveContainer" containerID="ab0f9e103eda50e270e4d3c8d44e0da13f31804a4a2cbfbb87b1c4842570cc08"
Sep 16 12:32:32.763361 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:32:32.763329    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0f9e103eda50e270e4d3c8d44e0da13f31804a4a2cbfbb87b1c4842570cc08\": container with ID starting with ab0f9e103eda50e270e4d3c8d44e0da13f31804a4a2cbfbb87b1c4842570cc08 not found: ID does not exist" containerID="ab0f9e103eda50e270e4d3c8d44e0da13f31804a4a2cbfbb87b1c4842570cc08"
Sep 16 12:32:32.763453 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.763367    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0f9e103eda50e270e4d3c8d44e0da13f31804a4a2cbfbb87b1c4842570cc08"} err="failed to get container status \"ab0f9e103eda50e270e4d3c8d44e0da13f31804a4a2cbfbb87b1c4842570cc08\": rpc error: code = NotFound desc = could not find container \"ab0f9e103eda50e270e4d3c8d44e0da13f31804a4a2cbfbb87b1c4842570cc08\": container with ID starting with ab0f9e103eda50e270e4d3c8d44e0da13f31804a4a2cbfbb87b1c4842570cc08 not found: ID does not exist"
Sep 16 12:32:32.774788 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.774760    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"]
Sep 16 12:32:32.778197 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:32.778176    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/auth-test-model-3be0c5-kserve-57b6dbddc9-h6lp8"]
Sep 16 12:32:33.215040 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:33.215006    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0741457e-9313-48d6-a2ce-0f12dd916445" path="/var/lib/kubelet/pods/0741457e-9313-48d6-a2ce-0f12dd916445/volumes"
Sep 16 12:32:33.215431 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:33.215416    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b5f731c-b509-4f45-a119-e7d4e520ac63" path="/var/lib/kubelet/pods/1b5f731c-b509-4f45-a119-e7d4e520ac63/volumes"
Sep 16 12:32:40.755836 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:40.755798    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"]
Sep 16 12:32:40.756288 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:40.756065    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft" podUID="cc6c4a13-2d36-4c6b-a3e7-0d67996e3103" containerName="maas-api" containerID="cri-o://af222f7f3eae91cb845ac7d03008d19508ada195008e825b3a4aae2159bbca95" gracePeriod=30
Sep 16 12:32:41.008635 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.008565    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:32:41.143933 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.143897    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-maas-api-tls\") pod \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\" (UID: \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\") "
Sep 16 12:32:41.144123 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.143949    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-openshift-service-ca\") pod \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\" (UID: \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\") "
Sep 16 12:32:41.144123 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.144009    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4m62\" (UniqueName: \"kubernetes.io/projected/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-kube-api-access-k4m62\") pod \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\" (UID: \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\") "
Sep 16 12:32:41.144123 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.144037    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-metrics-certs\") pod \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\" (UID: \"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103\") "
Sep 16 12:32:41.144382 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.144350    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc6c4a13-2d36-4c6b-a3e7-0d67996e3103" (UID: "cc6c4a13-2d36-4c6b-a3e7-0d67996e3103"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:32:41.146291 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.146260    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cc6c4a13-2d36-4c6b-a3e7-0d67996e3103" (UID: "cc6c4a13-2d36-4c6b-a3e7-0d67996e3103"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:32:41.146413 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.146369    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc6c4a13-2d36-4c6b-a3e7-0d67996e3103" (UID: "cc6c4a13-2d36-4c6b-a3e7-0d67996e3103"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:32:41.146463 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.146435    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-kube-api-access-k4m62" (OuterVolumeSpecName: "kube-api-access-k4m62") pod "cc6c4a13-2d36-4c6b-a3e7-0d67996e3103" (UID: "cc6c4a13-2d36-4c6b-a3e7-0d67996e3103"). InnerVolumeSpecName "kube-api-access-k4m62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:41.245189 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.245153    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:41.245189 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.245181    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:41.245189 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.245192    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:41.245413 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.245201    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4m62\" (UniqueName: \"kubernetes.io/projected/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103-kube-api-access-k4m62\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:32:41.637312 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.637275    2582 scope.go:117] "RemoveContainer" containerID="af222f7f3eae91cb845ac7d03008d19508ada195008e825b3a4aae2159bbca95"
Sep 16 12:32:41.802968 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.802927    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft" event={"ID":"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103","Type":"ContainerDied","Data":"af222f7f3eae91cb845ac7d03008d19508ada195008e825b3a4aae2159bbca95"}
Sep 16 12:32:41.803467 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.802984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft" event={"ID":"cc6c4a13-2d36-4c6b-a3e7-0d67996e3103","Type":"ContainerDied","Data":"0353c8b0a7b95ba270ac21ca4c5598f2886c361f44c9637196ffdd841880f554"}
Sep 16 12:32:41.803467 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.802989    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"
Sep 16 12:32:41.841868 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.841831    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"]
Sep 16 12:32:41.844548 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:41.844525    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f9133e-5fb8d865b-xwzft"]
Sep 16 12:32:43.212753 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:43.212702    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc6c4a13-2d36-4c6b-a3e7-0d67996e3103" path="/var/lib/kubelet/pods/cc6c4a13-2d36-4c6b-a3e7-0d67996e3103/volumes"
Sep 16 12:32:44.904797 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.904757    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"]
Sep 16 12:32:44.905371 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.905288    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="265104ae-4954-451b-ad38-934f9e35f843" containerName="test-curl-7567-1f1fdf"
Sep 16 12:32:44.905371 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.905309    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="265104ae-4954-451b-ad38-934f9e35f843" containerName="test-curl-7567-1f1fdf"
Sep 16 12:32:44.905371 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.905324    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06773e03-fd60-43ae-aa35-8dd3b5f6a665" containerName="maas-api"
Sep 16 12:32:44.905371 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.905334    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="06773e03-fd60-43ae-aa35-8dd3b5f6a665" containerName="maas-api"
Sep 16 12:32:44.905371 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.905350    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b5f731c-b509-4f45-a119-e7d4e520ac63" containerName="main"
Sep 16 12:32:44.905371 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.905358    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b5f731c-b509-4f45-a119-e7d4e520ac63" containerName="main"
Sep 16 12:32:44.905371 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.905373    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0741457e-9313-48d6-a2ce-0f12dd916445" containerName="main"
Sep 16 12:32:44.905977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.905381    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0741457e-9313-48d6-a2ce-0f12dd916445" containerName="main"
Sep 16 12:32:44.905977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.905392    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc6c4a13-2d36-4c6b-a3e7-0d67996e3103" containerName="maas-api"
Sep 16 12:32:44.905977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.905400    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc6c4a13-2d36-4c6b-a3e7-0d67996e3103" containerName="maas-api"
Sep 16 12:32:44.905977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.905506    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="265104ae-4954-451b-ad38-934f9e35f843" containerName="test-curl-7567-1f1fdf"
Sep 16 12:32:44.905977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.905519    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b5f731c-b509-4f45-a119-e7d4e520ac63" containerName="main"
Sep 16 12:32:44.905977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.905528    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc6c4a13-2d36-4c6b-a3e7-0d67996e3103" containerName="maas-api"
Sep 16 12:32:44.905977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.905541    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0741457e-9313-48d6-a2ce-0f12dd916445" containerName="main"
Sep 16 12:32:44.905977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.905548    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="06773e03-fd60-43ae-aa35-8dd3b5f6a665" containerName="maas-api"
Sep 16 12:32:44.908834 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.908810    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:44.912619 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.912597    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-72eee5\"/\"default-dockercfg-9d5h8\""
Sep 16 12:32:44.912775 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.912654    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-72eee5\"/\"openshift-service-ca.crt\""
Sep 16 12:32:44.913959 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.913939    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-72eee5\"/\"kube-root-ca.crt\""
Sep 16 12:32:44.914088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.913995    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-72eee5\"/\"sub-test-model-72eee5-kserve-self-signed-certs\""
Sep 16 12:32:44.923940 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:44.923911    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"]
Sep 16 12:32:45.082655 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.082621    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-dshm\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.082655 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.082658    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7nmc\" (UniqueName: \"kubernetes.io/projected/552585cf-2a8e-43a7-a9ce-9f424837e869-kube-api-access-p7nmc\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.082908 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.082689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552585cf-2a8e-43a7-a9ce-9f424837e869-tls-certs\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.082908 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.082792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-tmp-dir\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.082908 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.082865    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-home\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.082908 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.082902    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-model-cache\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.184234 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.184132    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-home\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.184234 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.184191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-model-cache\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.184234 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.184225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-dshm\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.184539 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.184245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7nmc\" (UniqueName: \"kubernetes.io/projected/552585cf-2a8e-43a7-a9ce-9f424837e869-kube-api-access-p7nmc\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.184539 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.184271    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552585cf-2a8e-43a7-a9ce-9f424837e869-tls-certs\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.184539 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.184297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-tmp-dir\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.184701 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.184611    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-home\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.184812 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.184710    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-tmp-dir\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.184812 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.184763    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-model-cache\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.186885 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.186863    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-dshm\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.187044 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.187026    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552585cf-2a8e-43a7-a9ce-9f424837e869-tls-certs\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.192879 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.192851    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7nmc\" (UniqueName: \"kubernetes.io/projected/552585cf-2a8e-43a7-a9ce-9f424837e869-kube-api-access-p7nmc\") pod \"sub-test-model-72eee5-kserve-7ddd958f69-q2fx4\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.218208 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.218176    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:45.490234 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.490205    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"]
Sep 16 12:32:45.491855 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:32:45.491826    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod552585cf_2a8e_43a7_a9ce_9f424837e869.slice/crio-3cfd7f52b7a66becd7b244ef984e194cad822e98c555740d4c05010c6483ddb7 WatchSource:0}: Error finding container 3cfd7f52b7a66becd7b244ef984e194cad822e98c555740d4c05010c6483ddb7: Status 404 returned error can't find the container with id 3cfd7f52b7a66becd7b244ef984e194cad822e98c555740d4c05010c6483ddb7
Sep 16 12:32:45.825104 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.825061    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4" event={"ID":"552585cf-2a8e-43a7-a9ce-9f424837e869","Type":"ContainerStarted","Data":"a0890605c378ffece6c78603f1b22e479b51fad7434e47e46a04f391f77d3422"}
Sep 16 12:32:45.825104 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.825107    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4" event={"ID":"552585cf-2a8e-43a7-a9ce-9f424837e869","Type":"ContainerStarted","Data":"3cfd7f52b7a66becd7b244ef984e194cad822e98c555740d4c05010c6483ddb7"}
Sep 16 12:32:45.842841 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:45.842788    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4" podStartSLOduration=1.842772533 podStartE2EDuration="1.842772533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:32:45.842061065 +0000 UTC m=+1925.092598617" watchObservedRunningTime="2026-09-16 12:32:45.842772533 +0000 UTC m=+1925.093310079"
Sep 16 12:32:55.219098 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:55.219067    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:55.219581 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:55.219110    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:55.232282 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:55.232258    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:32:55.889953 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:32:55.889921    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:33:10.005032 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.004996    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"]
Sep 16 12:33:10.008595 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.008575    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.012361 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.012325    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3be0c5\"/\"openshift-service-ca.crt\""
Sep 16 12:33:10.012517 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.012325    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3be0c5\"/\"sub-test-model-3be0c5-kserve-self-signed-certs\""
Sep 16 12:33:10.012517 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.012370    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3be0c5\"/\"kube-root-ca.crt\""
Sep 16 12:33:10.012517 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.012325    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3be0c5\"/\"default-dockercfg-c4qmb\""
Sep 16 12:33:10.018110 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.018086    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"]
Sep 16 12:33:10.109799 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.109758    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ec7f253-3dd9-494d-a99e-8909395cd27f-tls-certs\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.109983 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.109813    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-model-cache\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.109983 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.109844    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-tmp-dir\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.109983 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.109940    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-dshm\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.109983 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.109972    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt9l5\" (UniqueName: \"kubernetes.io/projected/0ec7f253-3dd9-494d-a99e-8909395cd27f-kube-api-access-xt9l5\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.110149 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.109998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-home\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.211281 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.211243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ec7f253-3dd9-494d-a99e-8909395cd27f-tls-certs\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.211281 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.211289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-model-cache\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.211574 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.211313    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-tmp-dir\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.211574 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.211370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-dshm\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.211574 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.211398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt9l5\" (UniqueName: \"kubernetes.io/projected/0ec7f253-3dd9-494d-a99e-8909395cd27f-kube-api-access-xt9l5\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.211574 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.211434    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-home\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.211845 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.211804    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-model-cache\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.211845 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.211825    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-home\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.211977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.211910    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-tmp-dir\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.213968 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.213938    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-dshm\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.214114 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.214095    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ec7f253-3dd9-494d-a99e-8909395cd27f-tls-certs\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.219572 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.219550    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt9l5\" (UniqueName: \"kubernetes.io/projected/0ec7f253-3dd9-494d-a99e-8909395cd27f-kube-api-access-xt9l5\") pod \"sub-test-model-3be0c5-kserve-69c684bcff-s6zbq\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.319581 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.319545    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:10.590573 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.590548    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"]
Sep 16 12:33:10.592419 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:33:10.592393    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ec7f253_3dd9_494d_a99e_8909395cd27f.slice/crio-ce59ec42f66365fa913cd969371e2cecd3c0526d75417850e68267d0eeadd10c WatchSource:0}: Error finding container ce59ec42f66365fa913cd969371e2cecd3c0526d75417850e68267d0eeadd10c: Status 404 returned error can't find the container with id ce59ec42f66365fa913cd969371e2cecd3c0526d75417850e68267d0eeadd10c
Sep 16 12:33:10.963383 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.963274    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq" event={"ID":"0ec7f253-3dd9-494d-a99e-8909395cd27f","Type":"ContainerStarted","Data":"f8bf9f45d0e36eb66e4f43a2d9f5577a033062c92d6af579f7441ff00614f30f"}
Sep 16 12:33:10.963383 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.963330    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq" event={"ID":"0ec7f253-3dd9-494d-a99e-8909395cd27f","Type":"ContainerStarted","Data":"ce59ec42f66365fa913cd969371e2cecd3c0526d75417850e68267d0eeadd10c"}
Sep 16 12:33:10.981116 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:10.981069    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq" podStartSLOduration=1.981052507 podStartE2EDuration="1.981052507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:33:10.978700751 +0000 UTC m=+1950.229238313" watchObservedRunningTime="2026-09-16 12:33:10.981052507 +0000 UTC m=+1950.231590407"
Sep 16 12:33:20.320518 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:20.320461    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:20.320518 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:20.320535    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:20.333229 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:20.333198    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:21.030765 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:21.030721    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:33:54.603334 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:54.603254    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"]
Sep 16 12:33:55.685186 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:55.685150    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"]
Sep 16 12:33:59.725648 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.725603    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"]
Sep 16 12:33:59.729078 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.729062    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.731877 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.731852    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-72eee5\"/\"rate-test-model-72eee5-kserve-self-signed-certs\""
Sep 16 12:33:59.738649 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.738621    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"]
Sep 16 12:33:59.879507 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.879467    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-home\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.879695 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.879517    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8jn2\" (UniqueName: \"kubernetes.io/projected/64d95808-6d36-4351-aa50-27b0a9682bb8-kube-api-access-h8jn2\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.879695 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.879572    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64d95808-6d36-4351-aa50-27b0a9682bb8-tls-certs\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.879695 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.879610    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-tmp-dir\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.879695 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.879630    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-dshm\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.879695 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.879662    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-model-cache\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.980864 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.980758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-model-cache\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.980864 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.980844    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-home\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.981102 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.980871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8jn2\" (UniqueName: \"kubernetes.io/projected/64d95808-6d36-4351-aa50-27b0a9682bb8-kube-api-access-h8jn2\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.981102 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.980892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64d95808-6d36-4351-aa50-27b0a9682bb8-tls-certs\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.981102 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.980914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-tmp-dir\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.981102 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.980939    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-dshm\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.981318 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.981268    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-model-cache\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.981318 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.981302    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-home\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.981432 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.981372    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-tmp-dir\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.983267 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.983250    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-dshm\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.983574 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.983557    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64d95808-6d36-4351-aa50-27b0a9682bb8-tls-certs\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:33:59.989758 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:33:59.989712    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8jn2\" (UniqueName: \"kubernetes.io/projected/64d95808-6d36-4351-aa50-27b0a9682bb8-kube-api-access-h8jn2\") pod \"rate-test-model-72eee5-kserve-7785d8f6c5-khm4v\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") " pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:34:00.039536 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:00.039498    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:34:00.316584 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:00.316555    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"]
Sep 16 12:34:00.318652 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:34:00.318624    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64d95808_6d36_4351_aa50_27b0a9682bb8.slice/crio-569e286d0b63c07ce130832056132a0bf0c32444c9289c3ad221b6ced8c90d80 WatchSource:0}: Error finding container 569e286d0b63c07ce130832056132a0bf0c32444c9289c3ad221b6ced8c90d80: Status 404 returned error can't find the container with id 569e286d0b63c07ce130832056132a0bf0c32444c9289c3ad221b6ced8c90d80
Sep 16 12:34:01.251884 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:01.251851    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v" event={"ID":"64d95808-6d36-4351-aa50-27b0a9682bb8","Type":"ContainerStarted","Data":"8f1fbcb8692c363a75a9baafb44ad21ae7253e8294cf8bc91cf8de9c655738c7"}
Sep 16 12:34:01.252281 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:01.251896    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v" event={"ID":"64d95808-6d36-4351-aa50-27b0a9682bb8","Type":"ContainerStarted","Data":"569e286d0b63c07ce130832056132a0bf0c32444c9289c3ad221b6ced8c90d80"}
Sep 16 12:34:01.269969 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:01.269915    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v" podStartSLOduration=2.269899688 podStartE2EDuration="2.269899688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:34:01.268716852 +0000 UTC m=+2000.519254402" watchObservedRunningTime="2026-09-16 12:34:01.269899688 +0000 UTC m=+2000.520437236"
Sep 16 12:34:09.622862 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:09.622811    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4" podUID="552585cf-2a8e-43a7-a9ce-9f424837e869" containerName="main" containerID="cri-o://a0890605c378ffece6c78603f1b22e479b51fad7434e47e46a04f391f77d3422" gracePeriod=45
Sep 16 12:34:09.885454 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:09.885393    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:34:10.040239 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.040192    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:34:10.040466 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.040422    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:34:10.053248 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.053218    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:34:10.081442 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.081410    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7nmc\" (UniqueName: \"kubernetes.io/projected/552585cf-2a8e-43a7-a9ce-9f424837e869-kube-api-access-p7nmc\") pod \"552585cf-2a8e-43a7-a9ce-9f424837e869\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") "
Sep 16 12:34:10.081442 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.081446    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-dshm\") pod \"552585cf-2a8e-43a7-a9ce-9f424837e869\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") "
Sep 16 12:34:10.081719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.081498    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-home\") pod \"552585cf-2a8e-43a7-a9ce-9f424837e869\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") "
Sep 16 12:34:10.081719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.081541    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-tmp-dir\") pod \"552585cf-2a8e-43a7-a9ce-9f424837e869\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") "
Sep 16 12:34:10.081719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.081564    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552585cf-2a8e-43a7-a9ce-9f424837e869-tls-certs\") pod \"552585cf-2a8e-43a7-a9ce-9f424837e869\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") "
Sep 16 12:34:10.081719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.081604    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-model-cache\") pod \"552585cf-2a8e-43a7-a9ce-9f424837e869\" (UID: \"552585cf-2a8e-43a7-a9ce-9f424837e869\") "
Sep 16 12:34:10.081988 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.081846    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-home" (OuterVolumeSpecName: "home") pod "552585cf-2a8e-43a7-a9ce-9f424837e869" (UID: "552585cf-2a8e-43a7-a9ce-9f424837e869"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:10.082057 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.082027    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-model-cache" (OuterVolumeSpecName: "model-cache") pod "552585cf-2a8e-43a7-a9ce-9f424837e869" (UID: "552585cf-2a8e-43a7-a9ce-9f424837e869"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:10.082057 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.082045    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:34:10.082166 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.082097    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "552585cf-2a8e-43a7-a9ce-9f424837e869" (UID: "552585cf-2a8e-43a7-a9ce-9f424837e869"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:10.083890 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.083867    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/552585cf-2a8e-43a7-a9ce-9f424837e869-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "552585cf-2a8e-43a7-a9ce-9f424837e869" (UID: "552585cf-2a8e-43a7-a9ce-9f424837e869"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:34:10.083890 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.083881    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552585cf-2a8e-43a7-a9ce-9f424837e869-kube-api-access-p7nmc" (OuterVolumeSpecName: "kube-api-access-p7nmc") pod "552585cf-2a8e-43a7-a9ce-9f424837e869" (UID: "552585cf-2a8e-43a7-a9ce-9f424837e869"). InnerVolumeSpecName "kube-api-access-p7nmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:34:10.084028 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.083931    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-dshm" (OuterVolumeSpecName: "dshm") pod "552585cf-2a8e-43a7-a9ce-9f424837e869" (UID: "552585cf-2a8e-43a7-a9ce-9f424837e869"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:10.183178 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.183082    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:34:10.183178 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.183114    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552585cf-2a8e-43a7-a9ce-9f424837e869-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:34:10.183178 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.183124    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:34:10.183178 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.183132    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7nmc\" (UniqueName: \"kubernetes.io/projected/552585cf-2a8e-43a7-a9ce-9f424837e869-kube-api-access-p7nmc\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:34:10.183178 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.183141    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552585cf-2a8e-43a7-a9ce-9f424837e869-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:34:10.305833 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.305787    2582 generic.go:356] "Generic (PLEG): container finished" podID="552585cf-2a8e-43a7-a9ce-9f424837e869" containerID="a0890605c378ffece6c78603f1b22e479b51fad7434e47e46a04f391f77d3422" exitCode=0
Sep 16 12:34:10.306033 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.305839    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"
Sep 16 12:34:10.306033 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.305868    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4" event={"ID":"552585cf-2a8e-43a7-a9ce-9f424837e869","Type":"ContainerDied","Data":"a0890605c378ffece6c78603f1b22e479b51fad7434e47e46a04f391f77d3422"}
Sep 16 12:34:10.306033 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.305918    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4" event={"ID":"552585cf-2a8e-43a7-a9ce-9f424837e869","Type":"ContainerDied","Data":"3cfd7f52b7a66becd7b244ef984e194cad822e98c555740d4c05010c6483ddb7"}
Sep 16 12:34:10.306033 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.305934    2582 scope.go:117] "RemoveContainer" containerID="a0890605c378ffece6c78603f1b22e479b51fad7434e47e46a04f391f77d3422"
Sep 16 12:34:10.316029 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.316009    2582 scope.go:117] "RemoveContainer" containerID="a0890605c378ffece6c78603f1b22e479b51fad7434e47e46a04f391f77d3422"
Sep 16 12:34:10.316352 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:34:10.316331    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0890605c378ffece6c78603f1b22e479b51fad7434e47e46a04f391f77d3422\": container with ID starting with a0890605c378ffece6c78603f1b22e479b51fad7434e47e46a04f391f77d3422 not found: ID does not exist" containerID="a0890605c378ffece6c78603f1b22e479b51fad7434e47e46a04f391f77d3422"
Sep 16 12:34:10.316415 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.316363    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0890605c378ffece6c78603f1b22e479b51fad7434e47e46a04f391f77d3422"} err="failed to get container status \"a0890605c378ffece6c78603f1b22e479b51fad7434e47e46a04f391f77d3422\": rpc error: code = NotFound desc = could not find container \"a0890605c378ffece6c78603f1b22e479b51fad7434e47e46a04f391f77d3422\": container with ID starting with a0890605c378ffece6c78603f1b22e479b51fad7434e47e46a04f391f77d3422 not found: ID does not exist"
Sep 16 12:34:10.318490 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.318470    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:34:10.331802 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.331768    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"]
Sep 16 12:34:10.334154 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.334128    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/sub-test-model-72eee5-kserve-7ddd958f69-q2fx4"]
Sep 16 12:34:10.705175 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.705129    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq" podUID="0ec7f253-3dd9-494d-a99e-8909395cd27f" containerName="main" containerID="cri-o://f8bf9f45d0e36eb66e4f43a2d9f5577a033062c92d6af579f7441ff00614f30f" gracePeriod=45
Sep 16 12:34:10.958346 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:10.958279    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:34:11.090336 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.090305    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt9l5\" (UniqueName: \"kubernetes.io/projected/0ec7f253-3dd9-494d-a99e-8909395cd27f-kube-api-access-xt9l5\") pod \"0ec7f253-3dd9-494d-a99e-8909395cd27f\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") "
Sep 16 12:34:11.090336 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.090339    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ec7f253-3dd9-494d-a99e-8909395cd27f-tls-certs\") pod \"0ec7f253-3dd9-494d-a99e-8909395cd27f\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") "
Sep 16 12:34:11.090624 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.090361    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-home\") pod \"0ec7f253-3dd9-494d-a99e-8909395cd27f\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") "
Sep 16 12:34:11.090624 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.090388    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-tmp-dir\") pod \"0ec7f253-3dd9-494d-a99e-8909395cd27f\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") "
Sep 16 12:34:11.090624 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.090510    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-model-cache\") pod \"0ec7f253-3dd9-494d-a99e-8909395cd27f\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") "
Sep 16 12:34:11.090624 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.090557    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-dshm\") pod \"0ec7f253-3dd9-494d-a99e-8909395cd27f\" (UID: \"0ec7f253-3dd9-494d-a99e-8909395cd27f\") "
Sep 16 12:34:11.090858 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.090637    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ec7f253-3dd9-494d-a99e-8909395cd27f" (UID: "0ec7f253-3dd9-494d-a99e-8909395cd27f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:11.090858 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.090650    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-home" (OuterVolumeSpecName: "home") pod "0ec7f253-3dd9-494d-a99e-8909395cd27f" (UID: "0ec7f253-3dd9-494d-a99e-8909395cd27f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:11.090858 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.090767    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ec7f253-3dd9-494d-a99e-8909395cd27f" (UID: "0ec7f253-3dd9-494d-a99e-8909395cd27f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:11.090977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.090945    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:34:11.090977 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.090966    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:34:11.091046 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.090980    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:34:11.092728 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.092704    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ec7f253-3dd9-494d-a99e-8909395cd27f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ec7f253-3dd9-494d-a99e-8909395cd27f" (UID: "0ec7f253-3dd9-494d-a99e-8909395cd27f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:34:11.092728 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.092710    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-dshm" (OuterVolumeSpecName: "dshm") pod "0ec7f253-3dd9-494d-a99e-8909395cd27f" (UID: "0ec7f253-3dd9-494d-a99e-8909395cd27f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:11.092915 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.092774    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec7f253-3dd9-494d-a99e-8909395cd27f-kube-api-access-xt9l5" (OuterVolumeSpecName: "kube-api-access-xt9l5") pod "0ec7f253-3dd9-494d-a99e-8909395cd27f" (UID: "0ec7f253-3dd9-494d-a99e-8909395cd27f"). InnerVolumeSpecName "kube-api-access-xt9l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:34:11.192146 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.192104    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt9l5\" (UniqueName: \"kubernetes.io/projected/0ec7f253-3dd9-494d-a99e-8909395cd27f-kube-api-access-xt9l5\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:34:11.192146 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.192136    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ec7f253-3dd9-494d-a99e-8909395cd27f-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:34:11.192146 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.192148    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ec7f253-3dd9-494d-a99e-8909395cd27f-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:34:11.212866 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.212783    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="552585cf-2a8e-43a7-a9ce-9f424837e869" path="/var/lib/kubelet/pods/552585cf-2a8e-43a7-a9ce-9f424837e869/volumes"
Sep 16 12:34:11.313846 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.313803    2582 generic.go:356] "Generic (PLEG): container finished" podID="0ec7f253-3dd9-494d-a99e-8909395cd27f" containerID="f8bf9f45d0e36eb66e4f43a2d9f5577a033062c92d6af579f7441ff00614f30f" exitCode=0
Sep 16 12:34:11.314071 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.313857    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"
Sep 16 12:34:11.314071 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.313900    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq" event={"ID":"0ec7f253-3dd9-494d-a99e-8909395cd27f","Type":"ContainerDied","Data":"f8bf9f45d0e36eb66e4f43a2d9f5577a033062c92d6af579f7441ff00614f30f"}
Sep 16 12:34:11.314071 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.313965    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq" event={"ID":"0ec7f253-3dd9-494d-a99e-8909395cd27f","Type":"ContainerDied","Data":"ce59ec42f66365fa913cd969371e2cecd3c0526d75417850e68267d0eeadd10c"}
Sep 16 12:34:11.314071 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.313991    2582 scope.go:117] "RemoveContainer" containerID="f8bf9f45d0e36eb66e4f43a2d9f5577a033062c92d6af579f7441ff00614f30f"
Sep 16 12:34:11.323543 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.323523    2582 scope.go:117] "RemoveContainer" containerID="f8bf9f45d0e36eb66e4f43a2d9f5577a033062c92d6af579f7441ff00614f30f"
Sep 16 12:34:11.323928 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:34:11.323906    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8bf9f45d0e36eb66e4f43a2d9f5577a033062c92d6af579f7441ff00614f30f\": container with ID starting with f8bf9f45d0e36eb66e4f43a2d9f5577a033062c92d6af579f7441ff00614f30f not found: ID does not exist" containerID="f8bf9f45d0e36eb66e4f43a2d9f5577a033062c92d6af579f7441ff00614f30f"
Sep 16 12:34:11.324012 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.323938    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8bf9f45d0e36eb66e4f43a2d9f5577a033062c92d6af579f7441ff00614f30f"} err="failed to get container status \"f8bf9f45d0e36eb66e4f43a2d9f5577a033062c92d6af579f7441ff00614f30f\": rpc error: code = NotFound desc = could not find container \"f8bf9f45d0e36eb66e4f43a2d9f5577a033062c92d6af579f7441ff00614f30f\": container with ID starting with f8bf9f45d0e36eb66e4f43a2d9f5577a033062c92d6af579f7441ff00614f30f not found: ID does not exist"
Sep 16 12:34:11.336429 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.336403    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"]
Sep 16 12:34:11.340460 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:11.340434    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/sub-test-model-3be0c5-kserve-69c684bcff-s6zbq"]
Sep 16 12:34:13.212955 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:13.212919    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ec7f253-3dd9-494d-a99e-8909395cd27f" path="/var/lib/kubelet/pods/0ec7f253-3dd9-494d-a99e-8909395cd27f/volumes"
Sep 16 12:34:18.120213 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.120169    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"]
Sep 16 12:34:18.120745 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.120714    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="552585cf-2a8e-43a7-a9ce-9f424837e869" containerName="main"
Sep 16 12:34:18.120804 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.120753    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="552585cf-2a8e-43a7-a9ce-9f424837e869" containerName="main"
Sep 16 12:34:18.120804 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.120777    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ec7f253-3dd9-494d-a99e-8909395cd27f" containerName="main"
Sep 16 12:34:18.120804 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.120786    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ec7f253-3dd9-494d-a99e-8909395cd27f" containerName="main"
Sep 16 12:34:18.120906 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.120887    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="552585cf-2a8e-43a7-a9ce-9f424837e869" containerName="main"
Sep 16 12:34:18.120906 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.120899    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ec7f253-3dd9-494d-a99e-8909395cd27f" containerName="main"
Sep 16 12:34:18.124354 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.124335    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.127548 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.127520    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3be0c5\"/\"default-dockercfg-c4qmb\""
Sep 16 12:34:18.127676 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.127563    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3be0c5\"/\"kube-root-ca.crt\""
Sep 16 12:34:18.128842 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.128819    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3be0c5\"/\"openshift-service-ca.crt\""
Sep 16 12:34:18.129062 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.129037    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3be0c5\"/\"rate-test-model-3be0c5-kserve-self-signed-certs\""
Sep 16 12:34:18.133587 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.133564    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"]
Sep 16 12:34:18.147883 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.147847    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-dshm\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.148013 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.147887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-home\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.148013 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.147957    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-tmp-dir\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.148013 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.147989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-model-cache\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.148167 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.148038    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr5q4\" (UniqueName: \"kubernetes.io/projected/ce5daa1b-e730-46d8-843e-70f9ee996b5e-kube-api-access-tr5q4\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.148167 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.148097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce5daa1b-e730-46d8-843e-70f9ee996b5e-tls-certs\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.249122 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.249065    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr5q4\" (UniqueName: \"kubernetes.io/projected/ce5daa1b-e730-46d8-843e-70f9ee996b5e-kube-api-access-tr5q4\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.249122 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.249128    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce5daa1b-e730-46d8-843e-70f9ee996b5e-tls-certs\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.249406 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.249196    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-dshm\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.249406 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.249214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-home\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.249406 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.249239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-tmp-dir\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.249406 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.249259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-model-cache\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.249702 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.249670    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-home\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.249702 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.249688    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-model-cache\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.249924 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.249788    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-tmp-dir\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.251661 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.251640    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-dshm\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.251960 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.251940    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce5daa1b-e730-46d8-843e-70f9ee996b5e-tls-certs\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.258397 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.258376    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr5q4\" (UniqueName: \"kubernetes.io/projected/ce5daa1b-e730-46d8-843e-70f9ee996b5e-kube-api-access-tr5q4\") pod \"rate-test-model-3be0c5-kserve-6545dffb75-gt6mt\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") " pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.436969 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.436888    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:18.710892 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:18.710857    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"]
Sep 16 12:34:18.712961 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:34:18.712926    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce5daa1b_e730_46d8_843e_70f9ee996b5e.slice/crio-062df63ae317e383304dd87b33ecbfbac7f3b9d3d43d2d9e7c6e705b6542bff4 WatchSource:0}: Error finding container 062df63ae317e383304dd87b33ecbfbac7f3b9d3d43d2d9e7c6e705b6542bff4: Status 404 returned error can't find the container with id 062df63ae317e383304dd87b33ecbfbac7f3b9d3d43d2d9e7c6e705b6542bff4
Sep 16 12:34:19.362015 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:19.361977    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt" event={"ID":"ce5daa1b-e730-46d8-843e-70f9ee996b5e","Type":"ContainerStarted","Data":"ff38c739b1a061435e92bde3e68adaca8808c720f30a5794ba2d4074e5bc331d"}
Sep 16 12:34:19.362015 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:19.362024    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt" event={"ID":"ce5daa1b-e730-46d8-843e-70f9ee996b5e","Type":"ContainerStarted","Data":"062df63ae317e383304dd87b33ecbfbac7f3b9d3d43d2d9e7c6e705b6542bff4"}
Sep 16 12:34:19.382959 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:19.382911    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt" podStartSLOduration=1.382895763 podStartE2EDuration="1.382895763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:34:19.380128209 +0000 UTC m=+2018.630665758" watchObservedRunningTime="2026-09-16 12:34:19.382895763 +0000 UTC m=+2018.633433346"
Sep 16 12:34:28.437640 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:28.437602    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:28.438191 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:28.437654    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:28.450401 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:28.450377    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:34:29.430549 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:34:29.430518    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:35:04.701898 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:04.701806    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"]
Sep 16 12:35:05.613220 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:05.612283    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"]
Sep 16 12:35:11.035267 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.035223    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"]
Sep 16 12:35:11.039704 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.039679    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:35:11.045330 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.045305    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3ba9e7-serving-cert\""
Sep 16 12:35:11.063130 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.063100    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"]
Sep 16 12:35:11.153332 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.153293    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85c37bf-5d62-488d-a79d-148798f00d07-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk\" (UID: \"a85c37bf-5d62-488d-a79d-148798f00d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:35:11.153527 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.153340    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhfwb\" (UniqueName: \"kubernetes.io/projected/a85c37bf-5d62-488d-a79d-148798f00d07-kube-api-access-jhfwb\") pod \"maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk\" (UID: \"a85c37bf-5d62-488d-a79d-148798f00d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:35:11.153527 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.153450    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85c37bf-5d62-488d-a79d-148798f00d07-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk\" (UID: \"a85c37bf-5d62-488d-a79d-148798f00d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:35:11.153527 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.153517    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85c37bf-5d62-488d-a79d-148798f00d07-metrics-certs\") pod \"maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk\" (UID: \"a85c37bf-5d62-488d-a79d-148798f00d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:35:11.254161 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.254122    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85c37bf-5d62-488d-a79d-148798f00d07-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk\" (UID: \"a85c37bf-5d62-488d-a79d-148798f00d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:35:11.254376 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.254180    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85c37bf-5d62-488d-a79d-148798f00d07-metrics-certs\") pod \"maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk\" (UID: \"a85c37bf-5d62-488d-a79d-148798f00d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:35:11.254376 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.254219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85c37bf-5d62-488d-a79d-148798f00d07-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk\" (UID: \"a85c37bf-5d62-488d-a79d-148798f00d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:35:11.254376 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.254243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhfwb\" (UniqueName: \"kubernetes.io/projected/a85c37bf-5d62-488d-a79d-148798f00d07-kube-api-access-jhfwb\") pod \"maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk\" (UID: \"a85c37bf-5d62-488d-a79d-148798f00d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:35:11.254869 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.254846    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85c37bf-5d62-488d-a79d-148798f00d07-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk\" (UID: \"a85c37bf-5d62-488d-a79d-148798f00d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:35:11.256892 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.256869    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85c37bf-5d62-488d-a79d-148798f00d07-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk\" (UID: \"a85c37bf-5d62-488d-a79d-148798f00d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:35:11.256991 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.256922    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85c37bf-5d62-488d-a79d-148798f00d07-metrics-certs\") pod \"maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk\" (UID: \"a85c37bf-5d62-488d-a79d-148798f00d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:35:11.273129 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.273104    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhfwb\" (UniqueName: \"kubernetes.io/projected/a85c37bf-5d62-488d-a79d-148798f00d07-kube-api-access-jhfwb\") pod \"maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk\" (UID: \"a85c37bf-5d62-488d-a79d-148798f00d07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:35:11.351794 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.351728    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:35:11.640458 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.640424    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"]
Sep 16 12:35:11.642428 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:35:11.642394    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85c37bf_5d62_488d_a79d_148798f00d07.slice/crio-4f79a400119397472fa0ea3319e3116265fb4080a0f546dc24442879b361f834 WatchSource:0}: Error finding container 4f79a400119397472fa0ea3319e3116265fb4080a0f546dc24442879b361f834: Status 404 returned error can't find the container with id 4f79a400119397472fa0ea3319e3116265fb4080a0f546dc24442879b361f834
Sep 16 12:35:11.654153 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:11.654125    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk" event={"ID":"a85c37bf-5d62-488d-a79d-148798f00d07","Type":"ContainerStarted","Data":"4f79a400119397472fa0ea3319e3116265fb4080a0f546dc24442879b361f834"}
Sep 16 12:35:12.661096 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:12.661055    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk" event={"ID":"a85c37bf-5d62-488d-a79d-148798f00d07","Type":"ContainerStarted","Data":"5f5f38382c6d84b2fe140eea89f1cabfcef704720c32e551720f14d646da4e21"}
Sep 16 12:35:12.661525 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:12.661159    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:35:12.681702 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:12.681649    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk" podStartSLOduration=1.277537602 podStartE2EDuration="1.681633175s" totalImagesPullingTime="404.095573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:35:12.678906208 +0000 UTC m=+2071.929443759" watchObservedRunningTime="2026-09-16 12:35:12.681633175 +0000 UTC m=+2071.932170721"
Sep 16 12:35:18.672148 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:18.672113    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:35:19.722296 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:19.722228    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v" podUID="64d95808-6d36-4351-aa50-27b0a9682bb8" containerName="main" containerID="cri-o://8f1fbcb8692c363a75a9baafb44ad21ae7253e8294cf8bc91cf8de9c655738c7" gracePeriod=45
Sep 16 12:35:19.974240 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:19.974186    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:35:20.036939 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.036896    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8jn2\" (UniqueName: \"kubernetes.io/projected/64d95808-6d36-4351-aa50-27b0a9682bb8-kube-api-access-h8jn2\") pod \"64d95808-6d36-4351-aa50-27b0a9682bb8\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") "
Sep 16 12:35:20.037139 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.037003    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-model-cache\") pod \"64d95808-6d36-4351-aa50-27b0a9682bb8\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") "
Sep 16 12:35:20.037139 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.037041    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-dshm\") pod \"64d95808-6d36-4351-aa50-27b0a9682bb8\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") "
Sep 16 12:35:20.037139 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.037071    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64d95808-6d36-4351-aa50-27b0a9682bb8-tls-certs\") pod \"64d95808-6d36-4351-aa50-27b0a9682bb8\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") "
Sep 16 12:35:20.037139 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.037121    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-tmp-dir\") pod \"64d95808-6d36-4351-aa50-27b0a9682bb8\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") "
Sep 16 12:35:20.037348 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.037145    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-home\") pod \"64d95808-6d36-4351-aa50-27b0a9682bb8\" (UID: \"64d95808-6d36-4351-aa50-27b0a9682bb8\") "
Sep 16 12:35:20.037348 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.037287    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-model-cache" (OuterVolumeSpecName: "model-cache") pod "64d95808-6d36-4351-aa50-27b0a9682bb8" (UID: "64d95808-6d36-4351-aa50-27b0a9682bb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:20.037442 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.037361    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64d95808-6d36-4351-aa50-27b0a9682bb8" (UID: "64d95808-6d36-4351-aa50-27b0a9682bb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:20.037513 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.037463    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-home" (OuterVolumeSpecName: "home") pod "64d95808-6d36-4351-aa50-27b0a9682bb8" (UID: "64d95808-6d36-4351-aa50-27b0a9682bb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:20.037570 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.037518    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:35:20.037570 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.037531    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:35:20.039287 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.039267    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-dshm" (OuterVolumeSpecName: "dshm") pod "64d95808-6d36-4351-aa50-27b0a9682bb8" (UID: "64d95808-6d36-4351-aa50-27b0a9682bb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:20.039405 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.039283    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d95808-6d36-4351-aa50-27b0a9682bb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64d95808-6d36-4351-aa50-27b0a9682bb8" (UID: "64d95808-6d36-4351-aa50-27b0a9682bb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:35:20.039405 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.039369    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d95808-6d36-4351-aa50-27b0a9682bb8-kube-api-access-h8jn2" (OuterVolumeSpecName: "kube-api-access-h8jn2") pod "64d95808-6d36-4351-aa50-27b0a9682bb8" (UID: "64d95808-6d36-4351-aa50-27b0a9682bb8"). InnerVolumeSpecName "kube-api-access-h8jn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:35:20.138078 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.138040    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:35:20.138078 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.138072    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64d95808-6d36-4351-aa50-27b0a9682bb8-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:35:20.138304 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.138094    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64d95808-6d36-4351-aa50-27b0a9682bb8-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:35:20.138304 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.138107    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8jn2\" (UniqueName: \"kubernetes.io/projected/64d95808-6d36-4351-aa50-27b0a9682bb8-kube-api-access-h8jn2\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:35:20.633546 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.633499    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt" podUID="ce5daa1b-e730-46d8-843e-70f9ee996b5e" containerName="main" containerID="cri-o://ff38c739b1a061435e92bde3e68adaca8808c720f30a5794ba2d4074e5bc331d" gracePeriod=45
Sep 16 12:35:20.707544 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.707500    2582 generic.go:356] "Generic (PLEG): container finished" podID="64d95808-6d36-4351-aa50-27b0a9682bb8" containerID="8f1fbcb8692c363a75a9baafb44ad21ae7253e8294cf8bc91cf8de9c655738c7" exitCode=0
Sep 16 12:35:20.707544 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.707542    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"
Sep 16 12:35:20.707859 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.707542    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v" event={"ID":"64d95808-6d36-4351-aa50-27b0a9682bb8","Type":"ContainerDied","Data":"8f1fbcb8692c363a75a9baafb44ad21ae7253e8294cf8bc91cf8de9c655738c7"}
Sep 16 12:35:20.707859 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.707676    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v" event={"ID":"64d95808-6d36-4351-aa50-27b0a9682bb8","Type":"ContainerDied","Data":"569e286d0b63c07ce130832056132a0bf0c32444c9289c3ad221b6ced8c90d80"}
Sep 16 12:35:20.707859 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.707706    2582 scope.go:117] "RemoveContainer" containerID="8f1fbcb8692c363a75a9baafb44ad21ae7253e8294cf8bc91cf8de9c655738c7"
Sep 16 12:35:20.730070 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.730037    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"]
Sep 16 12:35:20.734078 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.734035    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-72eee5/rate-test-model-72eee5-kserve-7785d8f6c5-khm4v"]
Sep 16 12:35:20.746120 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.745709    2582 scope.go:117] "RemoveContainer" containerID="8f1fbcb8692c363a75a9baafb44ad21ae7253e8294cf8bc91cf8de9c655738c7"
Sep 16 12:35:20.746669 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:35:20.746574    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f1fbcb8692c363a75a9baafb44ad21ae7253e8294cf8bc91cf8de9c655738c7\": container with ID starting with 8f1fbcb8692c363a75a9baafb44ad21ae7253e8294cf8bc91cf8de9c655738c7 not found: ID does not exist" containerID="8f1fbcb8692c363a75a9baafb44ad21ae7253e8294cf8bc91cf8de9c655738c7"
Sep 16 12:35:20.746669 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.746620    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f1fbcb8692c363a75a9baafb44ad21ae7253e8294cf8bc91cf8de9c655738c7"} err="failed to get container status \"8f1fbcb8692c363a75a9baafb44ad21ae7253e8294cf8bc91cf8de9c655738c7\": rpc error: code = NotFound desc = could not find container \"8f1fbcb8692c363a75a9baafb44ad21ae7253e8294cf8bc91cf8de9c655738c7\": container with ID starting with 8f1fbcb8692c363a75a9baafb44ad21ae7253e8294cf8bc91cf8de9c655738c7 not found: ID does not exist"
Sep 16 12:35:20.890954 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:20.890891    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:35:21.047560 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.047521    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-home\") pod \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") "
Sep 16 12:35:21.047808 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.047570    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-tmp-dir\") pod \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") "
Sep 16 12:35:21.047808 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.047601    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce5daa1b-e730-46d8-843e-70f9ee996b5e-tls-certs\") pod \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") "
Sep 16 12:35:21.047808 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.047647    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-dshm\") pod \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") "
Sep 16 12:35:21.047808 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.047681    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-model-cache\") pod \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") "
Sep 16 12:35:21.047808 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.047768    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr5q4\" (UniqueName: \"kubernetes.io/projected/ce5daa1b-e730-46d8-843e-70f9ee996b5e-kube-api-access-tr5q4\") pod \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\" (UID: \"ce5daa1b-e730-46d8-843e-70f9ee996b5e\") "
Sep 16 12:35:21.048107 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.047919    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-home" (OuterVolumeSpecName: "home") pod "ce5daa1b-e730-46d8-843e-70f9ee996b5e" (UID: "ce5daa1b-e730-46d8-843e-70f9ee996b5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:21.048107 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.047937    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce5daa1b-e730-46d8-843e-70f9ee996b5e" (UID: "ce5daa1b-e730-46d8-843e-70f9ee996b5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:21.048107 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.048001    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce5daa1b-e730-46d8-843e-70f9ee996b5e" (UID: "ce5daa1b-e730-46d8-843e-70f9ee996b5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:21.048244 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.048107    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:35:21.048244 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.048121    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:35:21.048244 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.048130    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:35:21.050073 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.050051    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce5daa1b-e730-46d8-843e-70f9ee996b5e-kube-api-access-tr5q4" (OuterVolumeSpecName: "kube-api-access-tr5q4") pod "ce5daa1b-e730-46d8-843e-70f9ee996b5e" (UID: "ce5daa1b-e730-46d8-843e-70f9ee996b5e"). InnerVolumeSpecName "kube-api-access-tr5q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:35:21.050170 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.050099    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce5daa1b-e730-46d8-843e-70f9ee996b5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce5daa1b-e730-46d8-843e-70f9ee996b5e" (UID: "ce5daa1b-e730-46d8-843e-70f9ee996b5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:35:21.050170 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.050120    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-dshm" (OuterVolumeSpecName: "dshm") pod "ce5daa1b-e730-46d8-843e-70f9ee996b5e" (UID: "ce5daa1b-e730-46d8-843e-70f9ee996b5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:21.149361 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.149233    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce5daa1b-e730-46d8-843e-70f9ee996b5e-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:35:21.149361 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.149265    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce5daa1b-e730-46d8-843e-70f9ee996b5e-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:35:21.149361 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.149275    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr5q4\" (UniqueName: \"kubernetes.io/projected/ce5daa1b-e730-46d8-843e-70f9ee996b5e-kube-api-access-tr5q4\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:35:21.213123 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.213085    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64d95808-6d36-4351-aa50-27b0a9682bb8" path="/var/lib/kubelet/pods/64d95808-6d36-4351-aa50-27b0a9682bb8/volumes"
Sep 16 12:35:21.720920 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.720881    2582 generic.go:356] "Generic (PLEG): container finished" podID="ce5daa1b-e730-46d8-843e-70f9ee996b5e" containerID="ff38c739b1a061435e92bde3e68adaca8808c720f30a5794ba2d4074e5bc331d" exitCode=0
Sep 16 12:35:21.721137 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.720930    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"
Sep 16 12:35:21.721137 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.720974    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt" event={"ID":"ce5daa1b-e730-46d8-843e-70f9ee996b5e","Type":"ContainerDied","Data":"ff38c739b1a061435e92bde3e68adaca8808c720f30a5794ba2d4074e5bc331d"}
Sep 16 12:35:21.721137 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.721020    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt" event={"ID":"ce5daa1b-e730-46d8-843e-70f9ee996b5e","Type":"ContainerDied","Data":"062df63ae317e383304dd87b33ecbfbac7f3b9d3d43d2d9e7c6e705b6542bff4"}
Sep 16 12:35:21.721137 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.721037    2582 scope.go:117] "RemoveContainer" containerID="ff38c739b1a061435e92bde3e68adaca8808c720f30a5794ba2d4074e5bc331d"
Sep 16 12:35:21.730187 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.730168    2582 scope.go:117] "RemoveContainer" containerID="ff38c739b1a061435e92bde3e68adaca8808c720f30a5794ba2d4074e5bc331d"
Sep 16 12:35:21.730497 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:35:21.730477    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff38c739b1a061435e92bde3e68adaca8808c720f30a5794ba2d4074e5bc331d\": container with ID starting with ff38c739b1a061435e92bde3e68adaca8808c720f30a5794ba2d4074e5bc331d not found: ID does not exist" containerID="ff38c739b1a061435e92bde3e68adaca8808c720f30a5794ba2d4074e5bc331d"
Sep 16 12:35:21.730548 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.730506    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff38c739b1a061435e92bde3e68adaca8808c720f30a5794ba2d4074e5bc331d"} err="failed to get container status \"ff38c739b1a061435e92bde3e68adaca8808c720f30a5794ba2d4074e5bc331d\": rpc error: code = NotFound desc = could not find container \"ff38c739b1a061435e92bde3e68adaca8808c720f30a5794ba2d4074e5bc331d\": container with ID starting with ff38c739b1a061435e92bde3e68adaca8808c720f30a5794ba2d4074e5bc331d not found: ID does not exist"
Sep 16 12:35:21.739388 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.739363    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"]
Sep 16 12:35:21.742948 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:21.742922    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3be0c5/rate-test-model-3be0c5-kserve-6545dffb75-gt6mt"]
Sep 16 12:35:23.212786 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:23.212729    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce5daa1b-e730-46d8-843e-70f9ee996b5e" path="/var/lib/kubelet/pods/ce5daa1b-e730-46d8-843e-70f9ee996b5e/volumes"
Sep 16 12:35:40.592256 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.592208    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"]
Sep 16 12:35:40.592935 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.592573    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64d95808-6d36-4351-aa50-27b0a9682bb8" containerName="main"
Sep 16 12:35:40.592935 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.592584    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="64d95808-6d36-4351-aa50-27b0a9682bb8" containerName="main"
Sep 16 12:35:40.592935 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.592600    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce5daa1b-e730-46d8-843e-70f9ee996b5e" containerName="main"
Sep 16 12:35:40.592935 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.592606    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce5daa1b-e730-46d8-843e-70f9ee996b5e" containerName="main"
Sep 16 12:35:40.592935 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.592670    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="64d95808-6d36-4351-aa50-27b0a9682bb8" containerName="main"
Sep 16 12:35:40.592935 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.592681    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce5daa1b-e730-46d8-843e-70f9ee996b5e" containerName="main"
Sep 16 12:35:40.597374 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.597340    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:35:40.600078 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.600048    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-43ef83-serving-cert\""
Sep 16 12:35:40.609439 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.609399    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"]
Sep 16 12:35:40.631375 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.631335    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52aefc18-18b2-4998-aaf9-f78d6708236b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8\" (UID: \"52aefc18-18b2-4998-aaf9-f78d6708236b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:35:40.631571 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.631399    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52aefc18-18b2-4998-aaf9-f78d6708236b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8\" (UID: \"52aefc18-18b2-4998-aaf9-f78d6708236b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:35:40.631571 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.631452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-788tc\" (UniqueName: \"kubernetes.io/projected/52aefc18-18b2-4998-aaf9-f78d6708236b-kube-api-access-788tc\") pod \"maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8\" (UID: \"52aefc18-18b2-4998-aaf9-f78d6708236b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:35:40.631571 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.631514    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52aefc18-18b2-4998-aaf9-f78d6708236b-metrics-certs\") pod \"maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8\" (UID: \"52aefc18-18b2-4998-aaf9-f78d6708236b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:35:40.732308 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.732258    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52aefc18-18b2-4998-aaf9-f78d6708236b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8\" (UID: \"52aefc18-18b2-4998-aaf9-f78d6708236b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:35:40.732308 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.732312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-788tc\" (UniqueName: \"kubernetes.io/projected/52aefc18-18b2-4998-aaf9-f78d6708236b-kube-api-access-788tc\") pod \"maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8\" (UID: \"52aefc18-18b2-4998-aaf9-f78d6708236b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:35:40.732587 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.732334    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52aefc18-18b2-4998-aaf9-f78d6708236b-metrics-certs\") pod \"maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8\" (UID: \"52aefc18-18b2-4998-aaf9-f78d6708236b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:35:40.732587 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.732421    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52aefc18-18b2-4998-aaf9-f78d6708236b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8\" (UID: \"52aefc18-18b2-4998-aaf9-f78d6708236b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:35:40.732587 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:35:40.732535    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-serving-cert: secret "maas-api-e2e-ipp-b-43ef83-serving-cert" not found
Sep 16 12:35:40.732812 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:35:40.732608    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52aefc18-18b2-4998-aaf9-f78d6708236b-maas-api-tls podName:52aefc18-18b2-4998-aaf9-f78d6708236b nodeName:}" failed. No retries permitted until 2026-09-16 12:35:41.232589253 +0000 UTC m=+2100.483126797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52aefc18-18b2-4998-aaf9-f78d6708236b-maas-api-tls") pod "maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8" (UID: "52aefc18-18b2-4998-aaf9-f78d6708236b") : secret "maas-api-e2e-ipp-b-43ef83-serving-cert" not found
Sep 16 12:35:40.733463 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.733414    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52aefc18-18b2-4998-aaf9-f78d6708236b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8\" (UID: \"52aefc18-18b2-4998-aaf9-f78d6708236b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:35:40.735101 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.735076    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52aefc18-18b2-4998-aaf9-f78d6708236b-metrics-certs\") pod \"maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8\" (UID: \"52aefc18-18b2-4998-aaf9-f78d6708236b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:35:40.741007 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:40.740984    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-788tc\" (UniqueName: \"kubernetes.io/projected/52aefc18-18b2-4998-aaf9-f78d6708236b-kube-api-access-788tc\") pod \"maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8\" (UID: \"52aefc18-18b2-4998-aaf9-f78d6708236b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:35:41.239149 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:41.239113    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52aefc18-18b2-4998-aaf9-f78d6708236b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8\" (UID: \"52aefc18-18b2-4998-aaf9-f78d6708236b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:35:41.241591 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:41.241555    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52aefc18-18b2-4998-aaf9-f78d6708236b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8\" (UID: \"52aefc18-18b2-4998-aaf9-f78d6708236b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:35:41.510984 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:41.510897    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:35:41.783648 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:41.783619    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"]
Sep 16 12:35:41.786384 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:35:41.786350    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52aefc18_18b2_4998_aaf9_f78d6708236b.slice/crio-04234fbcd700085df32f6654e37a78056c92dc6217182ae7ea297cf1ab35502a WatchSource:0}: Error finding container 04234fbcd700085df32f6654e37a78056c92dc6217182ae7ea297cf1ab35502a: Status 404 returned error can't find the container with id 04234fbcd700085df32f6654e37a78056c92dc6217182ae7ea297cf1ab35502a
Sep 16 12:35:41.788173 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:41.788154    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:35:41.829546 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:41.829511    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8" event={"ID":"52aefc18-18b2-4998-aaf9-f78d6708236b","Type":"ContainerStarted","Data":"04234fbcd700085df32f6654e37a78056c92dc6217182ae7ea297cf1ab35502a"}
Sep 16 12:35:42.837018 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:42.836976    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8" event={"ID":"52aefc18-18b2-4998-aaf9-f78d6708236b","Type":"ContainerStarted","Data":"e2491ecdf28ff55e715e8c5ca197534e6c31bae487471abf8090889670415070"}
Sep 16 12:35:42.837439 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:42.837192    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:35:42.855477 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:42.855425    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8" podStartSLOduration=2.428642013 podStartE2EDuration="2.855410454s" totalImagesPullingTime="426.768441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:35:42.854201263 +0000 UTC m=+2102.104738811" watchObservedRunningTime="2026-09-16 12:35:42.855410454 +0000 UTC m=+2102.105948004"
Sep 16 12:35:48.847618 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:35:48.847587    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:36:31.593373 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.593330    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"]
Sep 16 12:36:31.599025 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.598985    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.606888 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.605556    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3ba9e7\"/\"default-dockercfg-f8wx6\""
Sep 16 12:36:31.606888 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.605840    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3ba9e7\"/\"ipp-route-3ba9e7-kserve-self-signed-certs\""
Sep 16 12:36:31.606888 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.606128    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3ba9e7\"/\"kube-root-ca.crt\""
Sep 16 12:36:31.606888 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.606215    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3ba9e7\"/\"openshift-service-ca.crt\""
Sep 16 12:36:31.610017 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.609990    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"]
Sep 16 12:36:31.707893 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.707855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-model-cache\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.708125 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.707943    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-tmp-dir\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.708125 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.707976    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-dshm\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.708125 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.708066    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4t8k\" (UniqueName: \"kubernetes.io/projected/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-kube-api-access-j4t8k\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.708125 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.708097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-home\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.708125 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.708120    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-tls-certs\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.808646 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.808599    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4t8k\" (UniqueName: \"kubernetes.io/projected/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-kube-api-access-j4t8k\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.808646 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.808645    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-home\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.808961 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.808789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-tls-certs\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.808961 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.808859    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-model-cache\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.808961 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.808923    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-tmp-dir\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.808961 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.808947    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-dshm\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.809188 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.809079    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-home\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.809238 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.809208    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-model-cache\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.809295 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.809275    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-tmp-dir\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.811248 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.811228    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-dshm\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.811455 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.811437    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-tls-certs\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.820861 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.820834    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4t8k\" (UniqueName: \"kubernetes.io/projected/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-kube-api-access-j4t8k\") pod \"ipp-route-3ba9e7-kserve-79db667f6c-l56cx\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") " pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:31.913964 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:31.913869    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:32.185579 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:32.185548    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"]
Sep 16 12:36:32.186824 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:36:32.186765    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a02beec_b5a1_49c3_9b16_1c5bb41bea7b.slice/crio-1132a3f09034dc5229f24ec3970409684255ccdb172cf1eab6366cf4a365e53d WatchSource:0}: Error finding container 1132a3f09034dc5229f24ec3970409684255ccdb172cf1eab6366cf4a365e53d: Status 404 returned error can't find the container with id 1132a3f09034dc5229f24ec3970409684255ccdb172cf1eab6366cf4a365e53d
Sep 16 12:36:33.108941 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:33.108902    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx" event={"ID":"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b","Type":"ContainerStarted","Data":"1e07a19464060da4703ee37824375c29375bde7d15f0967210b50640b1c71487"}
Sep 16 12:36:33.109378 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:33.108955    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx" event={"ID":"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b","Type":"ContainerStarted","Data":"1132a3f09034dc5229f24ec3970409684255ccdb172cf1eab6366cf4a365e53d"}
Sep 16 12:36:33.142187 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:33.142133    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx" podStartSLOduration=2.142114601 podStartE2EDuration="2.142114601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:36:33.140443074 +0000 UTC m=+2152.390980657" watchObservedRunningTime="2026-09-16 12:36:33.142114601 +0000 UTC m=+2152.392652149"
Sep 16 12:36:41.914235 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:41.914197    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:41.914712 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:41.914254    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:41.927097 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:41.927067    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:36:42.170194 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:36:42.170110    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:37:01.692632 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:01.692593    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"]
Sep 16 12:37:01.697037 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:01.697013    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:01.699916 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:01.699887    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-6b0cae-serving-cert\""
Sep 16 12:37:01.706841 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:01.706815    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"]
Sep 16 12:37:01.790467 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:01.790431    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whqj7\" (UniqueName: \"kubernetes.io/projected/77e635ca-c237-4ef5-b27d-ac6536dee16e-kube-api-access-whqj7\") pod \"maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg\" (UID: \"77e635ca-c237-4ef5-b27d-ac6536dee16e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:01.790679 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:01.790513    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77e635ca-c237-4ef5-b27d-ac6536dee16e-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg\" (UID: \"77e635ca-c237-4ef5-b27d-ac6536dee16e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:01.790795 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:01.790672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77e635ca-c237-4ef5-b27d-ac6536dee16e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg\" (UID: \"77e635ca-c237-4ef5-b27d-ac6536dee16e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:01.790795 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:01.790703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77e635ca-c237-4ef5-b27d-ac6536dee16e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg\" (UID: \"77e635ca-c237-4ef5-b27d-ac6536dee16e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:01.891544 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:01.891499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77e635ca-c237-4ef5-b27d-ac6536dee16e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg\" (UID: \"77e635ca-c237-4ef5-b27d-ac6536dee16e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:01.891544 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:01.891546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77e635ca-c237-4ef5-b27d-ac6536dee16e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg\" (UID: \"77e635ca-c237-4ef5-b27d-ac6536dee16e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:01.891879 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:01.891592    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whqj7\" (UniqueName: \"kubernetes.io/projected/77e635ca-c237-4ef5-b27d-ac6536dee16e-kube-api-access-whqj7\") pod \"maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg\" (UID: \"77e635ca-c237-4ef5-b27d-ac6536dee16e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:01.891879 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:01.891618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77e635ca-c237-4ef5-b27d-ac6536dee16e-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg\" (UID: \"77e635ca-c237-4ef5-b27d-ac6536dee16e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:01.892217 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:01.892192    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77e635ca-c237-4ef5-b27d-ac6536dee16e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg\" (UID: \"77e635ca-c237-4ef5-b27d-ac6536dee16e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:01.894269 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:01.894247    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77e635ca-c237-4ef5-b27d-ac6536dee16e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg\" (UID: \"77e635ca-c237-4ef5-b27d-ac6536dee16e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:01.894356 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:01.894317    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77e635ca-c237-4ef5-b27d-ac6536dee16e-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg\" (UID: \"77e635ca-c237-4ef5-b27d-ac6536dee16e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:01.900227 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:01.900198    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whqj7\" (UniqueName: \"kubernetes.io/projected/77e635ca-c237-4ef5-b27d-ac6536dee16e-kube-api-access-whqj7\") pod \"maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg\" (UID: \"77e635ca-c237-4ef5-b27d-ac6536dee16e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:02.009913 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:02.009824    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:02.285437 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:02.285406    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"]
Sep 16 12:37:02.286523 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:37:02.286493    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e635ca_c237_4ef5_b27d_ac6536dee16e.slice/crio-dad17d772541a9eb1b20b9e6c1e0d0ac3f1d29f4a468f5cc2450ce4cb787d429 WatchSource:0}: Error finding container dad17d772541a9eb1b20b9e6c1e0d0ac3f1d29f4a468f5cc2450ce4cb787d429: Status 404 returned error can't find the container with id dad17d772541a9eb1b20b9e6c1e0d0ac3f1d29f4a468f5cc2450ce4cb787d429
Sep 16 12:37:03.280274 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:03.280231    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg" event={"ID":"77e635ca-c237-4ef5-b27d-ac6536dee16e","Type":"ContainerStarted","Data":"c9527515c802522521a032fbae4d70b3a71d01ea2d5d2ed24ab6cf67f758ea4f"}
Sep 16 12:37:03.280806 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:03.280291    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg" event={"ID":"77e635ca-c237-4ef5-b27d-ac6536dee16e","Type":"ContainerStarted","Data":"dad17d772541a9eb1b20b9e6c1e0d0ac3f1d29f4a468f5cc2450ce4cb787d429"}
Sep 16 12:37:03.280806 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:03.280397    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:03.300465 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:03.300412    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg" podStartSLOduration=1.8689766749999999 podStartE2EDuration="2.300393796s" totalImagesPullingTime="431.417121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:37:03.299395249 +0000 UTC m=+2182.549932802" watchObservedRunningTime="2026-09-16 12:37:03.300393796 +0000 UTC m=+2182.550931346"
Sep 16 12:37:09.292077 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:09.292043    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:40.506043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:40.506001    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"]
Sep 16 12:37:40.506610 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:40.506230    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg" podUID="77e635ca-c237-4ef5-b27d-ac6536dee16e" containerName="maas-api" containerID="cri-o://c9527515c802522521a032fbae4d70b3a71d01ea2d5d2ed24ab6cf67f758ea4f" gracePeriod=30
Sep 16 12:37:40.758898 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:40.758829    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:40.764611 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:40.764590    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whqj7\" (UniqueName: \"kubernetes.io/projected/77e635ca-c237-4ef5-b27d-ac6536dee16e-kube-api-access-whqj7\") pod \"77e635ca-c237-4ef5-b27d-ac6536dee16e\" (UID: \"77e635ca-c237-4ef5-b27d-ac6536dee16e\") "
Sep 16 12:37:40.764682 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:40.764633    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77e635ca-c237-4ef5-b27d-ac6536dee16e-openshift-service-ca\") pod \"77e635ca-c237-4ef5-b27d-ac6536dee16e\" (UID: \"77e635ca-c237-4ef5-b27d-ac6536dee16e\") "
Sep 16 12:37:40.764803 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:40.764784    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77e635ca-c237-4ef5-b27d-ac6536dee16e-maas-api-tls\") pod \"77e635ca-c237-4ef5-b27d-ac6536dee16e\" (UID: \"77e635ca-c237-4ef5-b27d-ac6536dee16e\") "
Sep 16 12:37:40.764875 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:40.764834    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77e635ca-c237-4ef5-b27d-ac6536dee16e-metrics-certs\") pod \"77e635ca-c237-4ef5-b27d-ac6536dee16e\" (UID: \"77e635ca-c237-4ef5-b27d-ac6536dee16e\") "
Sep 16 12:37:40.764962 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:40.764943    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e635ca-c237-4ef5-b27d-ac6536dee16e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "77e635ca-c237-4ef5-b27d-ac6536dee16e" (UID: "77e635ca-c237-4ef5-b27d-ac6536dee16e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:37:40.765147 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:40.765119    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77e635ca-c237-4ef5-b27d-ac6536dee16e-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:37:40.766938 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:40.766914    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e635ca-c237-4ef5-b27d-ac6536dee16e-kube-api-access-whqj7" (OuterVolumeSpecName: "kube-api-access-whqj7") pod "77e635ca-c237-4ef5-b27d-ac6536dee16e" (UID: "77e635ca-c237-4ef5-b27d-ac6536dee16e"). InnerVolumeSpecName "kube-api-access-whqj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:37:40.767043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:40.766965    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77e635ca-c237-4ef5-b27d-ac6536dee16e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "77e635ca-c237-4ef5-b27d-ac6536dee16e" (UID: "77e635ca-c237-4ef5-b27d-ac6536dee16e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:37:40.767043 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:40.767008    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77e635ca-c237-4ef5-b27d-ac6536dee16e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "77e635ca-c237-4ef5-b27d-ac6536dee16e" (UID: "77e635ca-c237-4ef5-b27d-ac6536dee16e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:37:40.866238 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:40.866202    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whqj7\" (UniqueName: \"kubernetes.io/projected/77e635ca-c237-4ef5-b27d-ac6536dee16e-kube-api-access-whqj7\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:37:40.866238 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:40.866231    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77e635ca-c237-4ef5-b27d-ac6536dee16e-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:37:40.866238 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:40.866241    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77e635ca-c237-4ef5-b27d-ac6536dee16e-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:37:41.506261 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:41.506173    2582 generic.go:356] "Generic (PLEG): container finished" podID="77e635ca-c237-4ef5-b27d-ac6536dee16e" containerID="c9527515c802522521a032fbae4d70b3a71d01ea2d5d2ed24ab6cf67f758ea4f" exitCode=0
Sep 16 12:37:41.506261 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:41.506255    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg" event={"ID":"77e635ca-c237-4ef5-b27d-ac6536dee16e","Type":"ContainerDied","Data":"c9527515c802522521a032fbae4d70b3a71d01ea2d5d2ed24ab6cf67f758ea4f"}
Sep 16 12:37:41.506717 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:41.506286    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"
Sep 16 12:37:41.506717 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:41.506320    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg" event={"ID":"77e635ca-c237-4ef5-b27d-ac6536dee16e","Type":"ContainerDied","Data":"dad17d772541a9eb1b20b9e6c1e0d0ac3f1d29f4a468f5cc2450ce4cb787d429"}
Sep 16 12:37:41.506717 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:41.506342    2582 scope.go:117] "RemoveContainer" containerID="c9527515c802522521a032fbae4d70b3a71d01ea2d5d2ed24ab6cf67f758ea4f"
Sep 16 12:37:41.516912 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:41.516889    2582 scope.go:117] "RemoveContainer" containerID="c9527515c802522521a032fbae4d70b3a71d01ea2d5d2ed24ab6cf67f758ea4f"
Sep 16 12:37:41.517206 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:37:41.517185    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9527515c802522521a032fbae4d70b3a71d01ea2d5d2ed24ab6cf67f758ea4f\": container with ID starting with c9527515c802522521a032fbae4d70b3a71d01ea2d5d2ed24ab6cf67f758ea4f not found: ID does not exist" containerID="c9527515c802522521a032fbae4d70b3a71d01ea2d5d2ed24ab6cf67f758ea4f"
Sep 16 12:37:41.517248 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:41.517218    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9527515c802522521a032fbae4d70b3a71d01ea2d5d2ed24ab6cf67f758ea4f"} err="failed to get container status \"c9527515c802522521a032fbae4d70b3a71d01ea2d5d2ed24ab6cf67f758ea4f\": rpc error: code = NotFound desc = could not find container \"c9527515c802522521a032fbae4d70b3a71d01ea2d5d2ed24ab6cf67f758ea4f\": container with ID starting with c9527515c802522521a032fbae4d70b3a71d01ea2d5d2ed24ab6cf67f758ea4f not found: ID does not exist"
Sep 16 12:37:41.523995 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:41.523973    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"]
Sep 16 12:37:41.527429 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:41.527403    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b0cae-6fdbfdd64d-5lbzg"]
Sep 16 12:37:43.212706 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:37:43.212669    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77e635ca-c237-4ef5-b27d-ac6536dee16e" path="/var/lib/kubelet/pods/77e635ca-c237-4ef5-b27d-ac6536dee16e/volumes"
Sep 16 12:38:06.397763 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.397605    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"]
Sep 16 12:38:06.398370 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.397864    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk" podUID="a85c37bf-5d62-488d-a79d-148798f00d07" containerName="maas-api" containerID="cri-o://5f5f38382c6d84b2fe140eea89f1cabfcef704720c32e551720f14d646da4e21" gracePeriod=30
Sep 16 12:38:06.641391 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.641358    2582 generic.go:356] "Generic (PLEG): container finished" podID="a85c37bf-5d62-488d-a79d-148798f00d07" containerID="5f5f38382c6d84b2fe140eea89f1cabfcef704720c32e551720f14d646da4e21" exitCode=0
Sep 16 12:38:06.641570 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.641432    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk" event={"ID":"a85c37bf-5d62-488d-a79d-148798f00d07","Type":"ContainerDied","Data":"5f5f38382c6d84b2fe140eea89f1cabfcef704720c32e551720f14d646da4e21"}
Sep 16 12:38:06.641570 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.641489    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk" event={"ID":"a85c37bf-5d62-488d-a79d-148798f00d07","Type":"ContainerDied","Data":"4f79a400119397472fa0ea3319e3116265fb4080a0f546dc24442879b361f834"}
Sep 16 12:38:06.641570 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.641506    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f79a400119397472fa0ea3319e3116265fb4080a0f546dc24442879b361f834"
Sep 16 12:38:06.647685 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.647655    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:38:06.715160 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.715068    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85c37bf-5d62-488d-a79d-148798f00d07-openshift-service-ca\") pod \"a85c37bf-5d62-488d-a79d-148798f00d07\" (UID: \"a85c37bf-5d62-488d-a79d-148798f00d07\") "
Sep 16 12:38:06.715323 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.715164    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85c37bf-5d62-488d-a79d-148798f00d07-metrics-certs\") pod \"a85c37bf-5d62-488d-a79d-148798f00d07\" (UID: \"a85c37bf-5d62-488d-a79d-148798f00d07\") "
Sep 16 12:38:06.715323 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.715200    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85c37bf-5d62-488d-a79d-148798f00d07-maas-api-tls\") pod \"a85c37bf-5d62-488d-a79d-148798f00d07\" (UID: \"a85c37bf-5d62-488d-a79d-148798f00d07\") "
Sep 16 12:38:06.715323 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.715277    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhfwb\" (UniqueName: \"kubernetes.io/projected/a85c37bf-5d62-488d-a79d-148798f00d07-kube-api-access-jhfwb\") pod \"a85c37bf-5d62-488d-a79d-148798f00d07\" (UID: \"a85c37bf-5d62-488d-a79d-148798f00d07\") "
Sep 16 12:38:06.715494 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.715465    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85c37bf-5d62-488d-a79d-148798f00d07-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a85c37bf-5d62-488d-a79d-148798f00d07" (UID: "a85c37bf-5d62-488d-a79d-148798f00d07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:38:06.715612 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.715589    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85c37bf-5d62-488d-a79d-148798f00d07-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:38:06.717648 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.717622    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85c37bf-5d62-488d-a79d-148798f00d07-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a85c37bf-5d62-488d-a79d-148798f00d07" (UID: "a85c37bf-5d62-488d-a79d-148798f00d07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:38:06.717776 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.717658    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85c37bf-5d62-488d-a79d-148798f00d07-kube-api-access-jhfwb" (OuterVolumeSpecName: "kube-api-access-jhfwb") pod "a85c37bf-5d62-488d-a79d-148798f00d07" (UID: "a85c37bf-5d62-488d-a79d-148798f00d07"). InnerVolumeSpecName "kube-api-access-jhfwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:38:06.717839 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.717775    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85c37bf-5d62-488d-a79d-148798f00d07-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a85c37bf-5d62-488d-a79d-148798f00d07" (UID: "a85c37bf-5d62-488d-a79d-148798f00d07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:38:06.817119 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.817085    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85c37bf-5d62-488d-a79d-148798f00d07-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:38:06.817119 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.817114    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85c37bf-5d62-488d-a79d-148798f00d07-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:38:06.817119 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:06.817123    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhfwb\" (UniqueName: \"kubernetes.io/projected/a85c37bf-5d62-488d-a79d-148798f00d07-kube-api-access-jhfwb\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:38:07.648204 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:07.648170    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"
Sep 16 12:38:07.665753 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:07.665702    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"]
Sep 16 12:38:07.671489 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:07.671461    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ba9e7-67f49b664c-5krmk"]
Sep 16 12:38:09.211985 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:09.211953    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a85c37bf-5d62-488d-a79d-148798f00d07" path="/var/lib/kubelet/pods/a85c37bf-5d62-488d-a79d-148798f00d07/volumes"
Sep 16 12:38:11.971556 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:11.971521    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"]
Sep 16 12:38:26.991778 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:26.991689    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx" podUID="9a02beec-b5a1-49c3-9b16-1c5bb41bea7b" containerName="main" containerID="cri-o://1e07a19464060da4703ee37824375c29375bde7d15f0967210b50640b1c71487" gracePeriod=45
Sep 16 12:38:27.252015 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.251955    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:38:27.298776 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.298720    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-tmp-dir\") pod \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") "
Sep 16 12:38:27.298776 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.298774    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-home\") pod \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") "
Sep 16 12:38:27.299073 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.298847    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-model-cache\") pod \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") "
Sep 16 12:38:27.299073 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.298879    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-tls-certs\") pod \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") "
Sep 16 12:38:27.299073 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.298914    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-dshm\") pod \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") "
Sep 16 12:38:27.299073 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.298953    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4t8k\" (UniqueName: \"kubernetes.io/projected/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-kube-api-access-j4t8k\") pod \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\" (UID: \"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b\") "
Sep 16 12:38:27.299303 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.299085    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9a02beec-b5a1-49c3-9b16-1c5bb41bea7b" (UID: "9a02beec-b5a1-49c3-9b16-1c5bb41bea7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:38:27.299303 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.299101    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-home" (OuterVolumeSpecName: "home") pod "9a02beec-b5a1-49c3-9b16-1c5bb41bea7b" (UID: "9a02beec-b5a1-49c3-9b16-1c5bb41bea7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:38:27.299303 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.299277    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:38:27.299303 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.299298    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:38:27.299511 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.299465    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-model-cache" (OuterVolumeSpecName: "model-cache") pod "9a02beec-b5a1-49c3-9b16-1c5bb41bea7b" (UID: "9a02beec-b5a1-49c3-9b16-1c5bb41bea7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:38:27.301588 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.301545    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a02beec-b5a1-49c3-9b16-1c5bb41bea7b" (UID: "9a02beec-b5a1-49c3-9b16-1c5bb41bea7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:38:27.301699 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.301679    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-kube-api-access-j4t8k" (OuterVolumeSpecName: "kube-api-access-j4t8k") pod "9a02beec-b5a1-49c3-9b16-1c5bb41bea7b" (UID: "9a02beec-b5a1-49c3-9b16-1c5bb41bea7b"). InnerVolumeSpecName "kube-api-access-j4t8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:38:27.301772 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.301716    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-dshm" (OuterVolumeSpecName: "dshm") pod "9a02beec-b5a1-49c3-9b16-1c5bb41bea7b" (UID: "9a02beec-b5a1-49c3-9b16-1c5bb41bea7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:38:27.400549 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.400505    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:38:27.400549 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.400539    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:38:27.400549 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.400548    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:38:27.400549 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.400556    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4t8k\" (UniqueName: \"kubernetes.io/projected/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b-kube-api-access-j4t8k\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:38:27.761644 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.761612    2582 generic.go:356] "Generic (PLEG): container finished" podID="9a02beec-b5a1-49c3-9b16-1c5bb41bea7b" containerID="1e07a19464060da4703ee37824375c29375bde7d15f0967210b50640b1c71487" exitCode=0
Sep 16 12:38:27.761847 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.761657    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"
Sep 16 12:38:27.761847 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.761662    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx" event={"ID":"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b","Type":"ContainerDied","Data":"1e07a19464060da4703ee37824375c29375bde7d15f0967210b50640b1c71487"}
Sep 16 12:38:27.761847 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.761695    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx" event={"ID":"9a02beec-b5a1-49c3-9b16-1c5bb41bea7b","Type":"ContainerDied","Data":"1132a3f09034dc5229f24ec3970409684255ccdb172cf1eab6366cf4a365e53d"}
Sep 16 12:38:27.761847 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.761712    2582 scope.go:117] "RemoveContainer" containerID="1e07a19464060da4703ee37824375c29375bde7d15f0967210b50640b1c71487"
Sep 16 12:38:27.771778 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.771763    2582 scope.go:117] "RemoveContainer" containerID="1e07a19464060da4703ee37824375c29375bde7d15f0967210b50640b1c71487"
Sep 16 12:38:27.772060 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:38:27.772039    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e07a19464060da4703ee37824375c29375bde7d15f0967210b50640b1c71487\": container with ID starting with 1e07a19464060da4703ee37824375c29375bde7d15f0967210b50640b1c71487 not found: ID does not exist" containerID="1e07a19464060da4703ee37824375c29375bde7d15f0967210b50640b1c71487"
Sep 16 12:38:27.772145 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.772071    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e07a19464060da4703ee37824375c29375bde7d15f0967210b50640b1c71487"} err="failed to get container status \"1e07a19464060da4703ee37824375c29375bde7d15f0967210b50640b1c71487\": rpc error: code = NotFound desc = could not find container \"1e07a19464060da4703ee37824375c29375bde7d15f0967210b50640b1c71487\": container with ID starting with 1e07a19464060da4703ee37824375c29375bde7d15f0967210b50640b1c71487 not found: ID does not exist"
Sep 16 12:38:27.787521 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.787406    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"]
Sep 16 12:38:27.789967 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:27.789944    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-3ba9e7/ipp-route-3ba9e7-kserve-79db667f6c-l56cx"]
Sep 16 12:38:29.212130 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:38:29.212099    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a02beec-b5a1-49c3-9b16-1c5bb41bea7b" path="/var/lib/kubelet/pods/9a02beec-b5a1-49c3-9b16-1c5bb41bea7b/volumes"
Sep 16 12:39:05.801101 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:05.801058    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"]
Sep 16 12:39:05.801646 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:05.801263    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8" podUID="52aefc18-18b2-4998-aaf9-f78d6708236b" containerName="maas-api" containerID="cri-o://e2491ecdf28ff55e715e8c5ca197534e6c31bae487471abf8090889670415070" gracePeriod=30
Sep 16 12:39:05.968497 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:05.968466    2582 generic.go:356] "Generic (PLEG): container finished" podID="52aefc18-18b2-4998-aaf9-f78d6708236b" containerID="e2491ecdf28ff55e715e8c5ca197534e6c31bae487471abf8090889670415070" exitCode=0
Sep 16 12:39:05.968681 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:05.968540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8" event={"ID":"52aefc18-18b2-4998-aaf9-f78d6708236b","Type":"ContainerDied","Data":"e2491ecdf28ff55e715e8c5ca197534e6c31bae487471abf8090889670415070"}
Sep 16 12:39:06.051921 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:06.051858    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:39:06.058548 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:06.058529    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52aefc18-18b2-4998-aaf9-f78d6708236b-metrics-certs\") pod \"52aefc18-18b2-4998-aaf9-f78d6708236b\" (UID: \"52aefc18-18b2-4998-aaf9-f78d6708236b\") "
Sep 16 12:39:06.058637 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:06.058562    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52aefc18-18b2-4998-aaf9-f78d6708236b-openshift-service-ca\") pod \"52aefc18-18b2-4998-aaf9-f78d6708236b\" (UID: \"52aefc18-18b2-4998-aaf9-f78d6708236b\") "
Sep 16 12:39:06.058637 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:06.058589    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-788tc\" (UniqueName: \"kubernetes.io/projected/52aefc18-18b2-4998-aaf9-f78d6708236b-kube-api-access-788tc\") pod \"52aefc18-18b2-4998-aaf9-f78d6708236b\" (UID: \"52aefc18-18b2-4998-aaf9-f78d6708236b\") "
Sep 16 12:39:06.058720 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:06.058638    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52aefc18-18b2-4998-aaf9-f78d6708236b-maas-api-tls\") pod \"52aefc18-18b2-4998-aaf9-f78d6708236b\" (UID: \"52aefc18-18b2-4998-aaf9-f78d6708236b\") "
Sep 16 12:39:06.058949 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:06.058924    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52aefc18-18b2-4998-aaf9-f78d6708236b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52aefc18-18b2-4998-aaf9-f78d6708236b" (UID: "52aefc18-18b2-4998-aaf9-f78d6708236b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:39:06.060899 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:06.060865    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52aefc18-18b2-4998-aaf9-f78d6708236b-kube-api-access-788tc" (OuterVolumeSpecName: "kube-api-access-788tc") pod "52aefc18-18b2-4998-aaf9-f78d6708236b" (UID: "52aefc18-18b2-4998-aaf9-f78d6708236b"). InnerVolumeSpecName "kube-api-access-788tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:39:06.060994 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:06.060899    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52aefc18-18b2-4998-aaf9-f78d6708236b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52aefc18-18b2-4998-aaf9-f78d6708236b" (UID: "52aefc18-18b2-4998-aaf9-f78d6708236b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:39:06.060994 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:06.060944    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52aefc18-18b2-4998-aaf9-f78d6708236b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "52aefc18-18b2-4998-aaf9-f78d6708236b" (UID: "52aefc18-18b2-4998-aaf9-f78d6708236b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:39:06.159936 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:06.159905    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52aefc18-18b2-4998-aaf9-f78d6708236b-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:39:06.159936 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:06.159934    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52aefc18-18b2-4998-aaf9-f78d6708236b-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:39:06.159936 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:06.159944    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52aefc18-18b2-4998-aaf9-f78d6708236b-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:39:06.160165 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:06.159952    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-788tc\" (UniqueName: \"kubernetes.io/projected/52aefc18-18b2-4998-aaf9-f78d6708236b-kube-api-access-788tc\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:39:06.975216 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:06.975177    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8" event={"ID":"52aefc18-18b2-4998-aaf9-f78d6708236b","Type":"ContainerDied","Data":"04234fbcd700085df32f6654e37a78056c92dc6217182ae7ea297cf1ab35502a"}
Sep 16 12:39:06.975216 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:06.975197    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"
Sep 16 12:39:06.975729 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:06.975251    2582 scope.go:117] "RemoveContainer" containerID="e2491ecdf28ff55e715e8c5ca197534e6c31bae487471abf8090889670415070"
Sep 16 12:39:07.003119 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:07.003083    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"]
Sep 16 12:39:07.006620 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:07.006594    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-43ef83-66bc7dc678-tvwt8"]
Sep 16 12:39:07.212681 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:07.212639    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52aefc18-18b2-4998-aaf9-f78d6708236b" path="/var/lib/kubelet/pods/52aefc18-18b2-4998-aaf9-f78d6708236b/volumes"
Sep 16 12:39:23.485849 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:23.485811    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"]
Sep 16 12:39:23.486356 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:23.486029    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg" podUID="7989b85b-a710-40ad-9b0b-6ab2de094f14" containerName="maas-api" containerID="cri-o://58d85d2be73c94358485398cbbf720d8c68fa829fa87902bd729bb0c33801f2e" gracePeriod=30
Sep 16 12:39:23.725997 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:23.725972    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:39:23.818309 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:23.818214    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7989b85b-a710-40ad-9b0b-6ab2de094f14-maas-api-tls\") pod \"7989b85b-a710-40ad-9b0b-6ab2de094f14\" (UID: \"7989b85b-a710-40ad-9b0b-6ab2de094f14\") "
Sep 16 12:39:23.818309 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:23.818300    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7989b85b-a710-40ad-9b0b-6ab2de094f14-openshift-service-ca\") pod \"7989b85b-a710-40ad-9b0b-6ab2de094f14\" (UID: \"7989b85b-a710-40ad-9b0b-6ab2de094f14\") "
Sep 16 12:39:23.818510 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:23.818339    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btv7s\" (UniqueName: \"kubernetes.io/projected/7989b85b-a710-40ad-9b0b-6ab2de094f14-kube-api-access-btv7s\") pod \"7989b85b-a710-40ad-9b0b-6ab2de094f14\" (UID: \"7989b85b-a710-40ad-9b0b-6ab2de094f14\") "
Sep 16 12:39:23.818510 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:23.818382    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7989b85b-a710-40ad-9b0b-6ab2de094f14-metrics-certs\") pod \"7989b85b-a710-40ad-9b0b-6ab2de094f14\" (UID: \"7989b85b-a710-40ad-9b0b-6ab2de094f14\") "
Sep 16 12:39:23.818714 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:23.818681    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7989b85b-a710-40ad-9b0b-6ab2de094f14-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7989b85b-a710-40ad-9b0b-6ab2de094f14" (UID: "7989b85b-a710-40ad-9b0b-6ab2de094f14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:39:23.820707 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:23.820672    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7989b85b-a710-40ad-9b0b-6ab2de094f14-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7989b85b-a710-40ad-9b0b-6ab2de094f14" (UID: "7989b85b-a710-40ad-9b0b-6ab2de094f14"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:39:23.820827 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:23.820766    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7989b85b-a710-40ad-9b0b-6ab2de094f14-kube-api-access-btv7s" (OuterVolumeSpecName: "kube-api-access-btv7s") pod "7989b85b-a710-40ad-9b0b-6ab2de094f14" (UID: "7989b85b-a710-40ad-9b0b-6ab2de094f14"). InnerVolumeSpecName "kube-api-access-btv7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:39:23.820827 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:23.820802    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7989b85b-a710-40ad-9b0b-6ab2de094f14-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7989b85b-a710-40ad-9b0b-6ab2de094f14" (UID: "7989b85b-a710-40ad-9b0b-6ab2de094f14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:39:23.919435 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:23.919394    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7989b85b-a710-40ad-9b0b-6ab2de094f14-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:39:23.919435 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:23.919427    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7989b85b-a710-40ad-9b0b-6ab2de094f14-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:39:23.919435 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:23.919437    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btv7s\" (UniqueName: \"kubernetes.io/projected/7989b85b-a710-40ad-9b0b-6ab2de094f14-kube-api-access-btv7s\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:39:23.919435 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:23.919445    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7989b85b-a710-40ad-9b0b-6ab2de094f14-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:39:24.071131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:24.071099    2582 generic.go:356] "Generic (PLEG): container finished" podID="7989b85b-a710-40ad-9b0b-6ab2de094f14" containerID="58d85d2be73c94358485398cbbf720d8c68fa829fa87902bd729bb0c33801f2e" exitCode=0
Sep 16 12:39:24.071326 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:24.071145    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"
Sep 16 12:39:24.071326 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:24.071186    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg" event={"ID":"7989b85b-a710-40ad-9b0b-6ab2de094f14","Type":"ContainerDied","Data":"58d85d2be73c94358485398cbbf720d8c68fa829fa87902bd729bb0c33801f2e"}
Sep 16 12:39:24.071326 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:24.071253    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg" event={"ID":"7989b85b-a710-40ad-9b0b-6ab2de094f14","Type":"ContainerDied","Data":"e1270b9738c0bc085330c8a4b9d2323d88dd67d2ea2281383cf968c4fd030a81"}
Sep 16 12:39:24.071326 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:24.071283    2582 scope.go:117] "RemoveContainer" containerID="58d85d2be73c94358485398cbbf720d8c68fa829fa87902bd729bb0c33801f2e"
Sep 16 12:39:24.080993 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:24.080977    2582 scope.go:117] "RemoveContainer" containerID="58d85d2be73c94358485398cbbf720d8c68fa829fa87902bd729bb0c33801f2e"
Sep 16 12:39:24.081261 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:39:24.081241    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d85d2be73c94358485398cbbf720d8c68fa829fa87902bd729bb0c33801f2e\": container with ID starting with 58d85d2be73c94358485398cbbf720d8c68fa829fa87902bd729bb0c33801f2e not found: ID does not exist" containerID="58d85d2be73c94358485398cbbf720d8c68fa829fa87902bd729bb0c33801f2e"
Sep 16 12:39:24.081332 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:24.081268    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d85d2be73c94358485398cbbf720d8c68fa829fa87902bd729bb0c33801f2e"} err="failed to get container status \"58d85d2be73c94358485398cbbf720d8c68fa829fa87902bd729bb0c33801f2e\": rpc error: code = NotFound desc = could not find container \"58d85d2be73c94358485398cbbf720d8c68fa829fa87902bd729bb0c33801f2e\": container with ID starting with 58d85d2be73c94358485398cbbf720d8c68fa829fa87902bd729bb0c33801f2e not found: ID does not exist"
Sep 16 12:39:24.093492 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:24.093463    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"]
Sep 16 12:39:24.097165 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:24.097140    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-72eee5-54d4fb9584-2fqdg"]
Sep 16 12:39:25.212324 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:25.212288    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7989b85b-a710-40ad-9b0b-6ab2de094f14" path="/var/lib/kubelet/pods/7989b85b-a710-40ad-9b0b-6ab2de094f14/volumes"
Sep 16 12:39:41.145955 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:41.145873    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"]
Sep 16 12:39:41.146459 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:41.146177    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2" podUID="dde19631-63ed-4833-8a70-e3bd95007396" containerName="maas-api" containerID="cri-o://af2b07b389817ceb1183066a348ce4f7beeae71b5bb27640dc8c806577b2ea3b" gracePeriod=30
Sep 16 12:39:41.399071 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:41.399004    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:39:41.476042 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:41.476009    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q64xw\" (UniqueName: \"kubernetes.io/projected/dde19631-63ed-4833-8a70-e3bd95007396-kube-api-access-q64xw\") pod \"dde19631-63ed-4833-8a70-e3bd95007396\" (UID: \"dde19631-63ed-4833-8a70-e3bd95007396\") "
Sep 16 12:39:41.476042 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:41.476045    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dde19631-63ed-4833-8a70-e3bd95007396-openshift-service-ca\") pod \"dde19631-63ed-4833-8a70-e3bd95007396\" (UID: \"dde19631-63ed-4833-8a70-e3bd95007396\") "
Sep 16 12:39:41.476330 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:41.476125    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dde19631-63ed-4833-8a70-e3bd95007396-maas-api-tls\") pod \"dde19631-63ed-4833-8a70-e3bd95007396\" (UID: \"dde19631-63ed-4833-8a70-e3bd95007396\") "
Sep 16 12:39:41.476330 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:41.476144    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dde19631-63ed-4833-8a70-e3bd95007396-metrics-certs\") pod \"dde19631-63ed-4833-8a70-e3bd95007396\" (UID: \"dde19631-63ed-4833-8a70-e3bd95007396\") "
Sep 16 12:39:41.476474 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:41.476445    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dde19631-63ed-4833-8a70-e3bd95007396-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dde19631-63ed-4833-8a70-e3bd95007396" (UID: "dde19631-63ed-4833-8a70-e3bd95007396"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:39:41.478442 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:41.478412    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde19631-63ed-4833-8a70-e3bd95007396-kube-api-access-q64xw" (OuterVolumeSpecName: "kube-api-access-q64xw") pod "dde19631-63ed-4833-8a70-e3bd95007396" (UID: "dde19631-63ed-4833-8a70-e3bd95007396"). InnerVolumeSpecName "kube-api-access-q64xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:39:41.478571 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:41.478447    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dde19631-63ed-4833-8a70-e3bd95007396-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dde19631-63ed-4833-8a70-e3bd95007396" (UID: "dde19631-63ed-4833-8a70-e3bd95007396"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:39:41.478571 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:41.478476    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dde19631-63ed-4833-8a70-e3bd95007396-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dde19631-63ed-4833-8a70-e3bd95007396" (UID: "dde19631-63ed-4833-8a70-e3bd95007396"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:39:41.577324 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:41.577276    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dde19631-63ed-4833-8a70-e3bd95007396-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:39:41.577324 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:41.577312    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dde19631-63ed-4833-8a70-e3bd95007396-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:39:41.577324 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:41.577327    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q64xw\" (UniqueName: \"kubernetes.io/projected/dde19631-63ed-4833-8a70-e3bd95007396-kube-api-access-q64xw\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:39:41.577576 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:41.577340    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dde19631-63ed-4833-8a70-e3bd95007396-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:39:41.772182 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:41.772100    2582 scope.go:117] "RemoveContainer" containerID="af2b07b389817ceb1183066a348ce4f7beeae71b5bb27640dc8c806577b2ea3b"
Sep 16 12:39:42.177699 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:42.177669    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"
Sep 16 12:39:42.178136 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:42.177664    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2" event={"ID":"dde19631-63ed-4833-8a70-e3bd95007396","Type":"ContainerDied","Data":"af2b07b389817ceb1183066a348ce4f7beeae71b5bb27640dc8c806577b2ea3b"}
Sep 16 12:39:42.178136 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:42.177816    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2" event={"ID":"dde19631-63ed-4833-8a70-e3bd95007396","Type":"ContainerDied","Data":"89296db3cafd12ff0a926ca89b057e3a5fc3369546c233d226ea6e89338e99e7"}
Sep 16 12:39:42.221336 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:42.221296    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"]
Sep 16 12:39:42.226524 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:42.226495    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3be0c5-647fd4864-pd5g2"]
Sep 16 12:39:43.211646 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:39:43.211608    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dde19631-63ed-4833-8a70-e3bd95007396" path="/var/lib/kubelet/pods/dde19631-63ed-4833-8a70-e3bd95007396/volumes"
Sep 16 12:41:41.809685 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:41:41.809652    2582 scope.go:117] "RemoveContainer" containerID="5f5f38382c6d84b2fe140eea89f1cabfcef704720c32e551720f14d646da4e21"
Sep 16 12:44:38.713166 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713105    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"]
Sep 16 12:44:38.713719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713540    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e635ca-c237-4ef5-b27d-ac6536dee16e" containerName="maas-api"
Sep 16 12:44:38.713719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713563    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e635ca-c237-4ef5-b27d-ac6536dee16e" containerName="maas-api"
Sep 16 12:44:38.713719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713578    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7989b85b-a710-40ad-9b0b-6ab2de094f14" containerName="maas-api"
Sep 16 12:44:38.713719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713589    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7989b85b-a710-40ad-9b0b-6ab2de094f14" containerName="maas-api"
Sep 16 12:44:38.713719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713599    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a85c37bf-5d62-488d-a79d-148798f00d07" containerName="maas-api"
Sep 16 12:44:38.713719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713605    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a85c37bf-5d62-488d-a79d-148798f00d07" containerName="maas-api"
Sep 16 12:44:38.713719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713612    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dde19631-63ed-4833-8a70-e3bd95007396" containerName="maas-api"
Sep 16 12:44:38.713719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713618    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dde19631-63ed-4833-8a70-e3bd95007396" containerName="maas-api"
Sep 16 12:44:38.713719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713631    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a02beec-b5a1-49c3-9b16-1c5bb41bea7b" containerName="main"
Sep 16 12:44:38.713719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713636    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a02beec-b5a1-49c3-9b16-1c5bb41bea7b" containerName="main"
Sep 16 12:44:38.713719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713651    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52aefc18-18b2-4998-aaf9-f78d6708236b" containerName="maas-api"
Sep 16 12:44:38.713719 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713659    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="52aefc18-18b2-4998-aaf9-f78d6708236b" containerName="maas-api"
Sep 16 12:44:38.714215 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713778    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="77e635ca-c237-4ef5-b27d-ac6536dee16e" containerName="maas-api"
Sep 16 12:44:38.714215 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713791    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7989b85b-a710-40ad-9b0b-6ab2de094f14" containerName="maas-api"
Sep 16 12:44:38.714215 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713802    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a85c37bf-5d62-488d-a79d-148798f00d07" containerName="maas-api"
Sep 16 12:44:38.714215 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713813    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dde19631-63ed-4833-8a70-e3bd95007396" containerName="maas-api"
Sep 16 12:44:38.714215 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713822    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="52aefc18-18b2-4998-aaf9-f78d6708236b" containerName="maas-api"
Sep 16 12:44:38.714215 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.713828    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a02beec-b5a1-49c3-9b16-1c5bb41bea7b" containerName="main"
Sep 16 12:44:38.716826 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.716809    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.719868 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.719844    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e14ade35-kserve-self-signed-certs\""
Sep 16 12:44:38.730646 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.730614    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"]
Sep 16 12:44:38.759555 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.759529    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-home\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.759768 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.759592    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-model-cache\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.759768 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.759637    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3bbfb8-d292-4006-9853-b6ac862e49ff-tls-certs\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.759768 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.759676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-dshm\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.759768 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.759710    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkqrd\" (UniqueName: \"kubernetes.io/projected/6a3bbfb8-d292-4006-9853-b6ac862e49ff-kube-api-access-lkqrd\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.759966 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.759769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-tmp-dir\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.861218 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.861180    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-home\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.861418 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.861235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-model-cache\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.861418 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.861272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3bbfb8-d292-4006-9853-b6ac862e49ff-tls-certs\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.861418 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.861295    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-dshm\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.861418 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.861320    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkqrd\" (UniqueName: \"kubernetes.io/projected/6a3bbfb8-d292-4006-9853-b6ac862e49ff-kube-api-access-lkqrd\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.861418 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.861342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-tmp-dir\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.861709 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.861629    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-home\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.861807 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.861705    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-model-cache\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.861807 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.861788    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-tmp-dir\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.863746 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.863708    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-dshm\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.864066 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.864047    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3bbfb8-d292-4006-9853-b6ac862e49ff-tls-certs\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:38.869983 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:38.869959    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkqrd\" (UniqueName: \"kubernetes.io/projected/6a3bbfb8-d292-4006-9853-b6ac862e49ff-kube-api-access-lkqrd\") pod \"e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") " pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:39.026948 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:39.026853    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:39.302597 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:39.302571    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"]
Sep 16 12:44:39.304664 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:44:39.304632    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a3bbfb8_d292_4006_9853_b6ac862e49ff.slice/crio-a398e742f76c16b9d89ff1daeb4a2af265c123756a2f15f01d3d44cfc0af71f1 WatchSource:0}: Error finding container a398e742f76c16b9d89ff1daeb4a2af265c123756a2f15f01d3d44cfc0af71f1: Status 404 returned error can't find the container with id a398e742f76c16b9d89ff1daeb4a2af265c123756a2f15f01d3d44cfc0af71f1
Sep 16 12:44:39.756342 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:39.756300    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h" event={"ID":"6a3bbfb8-d292-4006-9853-b6ac862e49ff","Type":"ContainerStarted","Data":"cf14fea771e5419c6f4b70118be2cb19733d6e8d8d5bd0bf8b9ae3fb7544939b"}
Sep 16 12:44:39.756342 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:39.756349    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h" event={"ID":"6a3bbfb8-d292-4006-9853-b6ac862e49ff","Type":"ContainerStarted","Data":"a398e742f76c16b9d89ff1daeb4a2af265c123756a2f15f01d3d44cfc0af71f1"}
Sep 16 12:44:39.775129 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:39.775077    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h" podStartSLOduration=1.775062017 podStartE2EDuration="1.775062017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:44:39.773757465 +0000 UTC m=+2639.024295006" watchObservedRunningTime="2026-09-16 12:44:39.775062017 +0000 UTC m=+2639.025599624"
Sep 16 12:44:42.015294 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.015253    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"]
Sep 16 12:44:42.018970 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.018945    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.021543 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.021520    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e14ade35-kserve-self-signed-certs\""
Sep 16 12:44:42.029313 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.029288    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"]
Sep 16 12:44:42.090446 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.090412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvxmb\" (UniqueName: \"kubernetes.io/projected/073698f5-b519-4201-9be8-d06e6f69220c-kube-api-access-rvxmb\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.090653 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.090478    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-dshm\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.090653 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.090510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-home\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.090653 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.090536    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/073698f5-b519-4201-9be8-d06e6f69220c-tls-certs\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.090653 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.090581    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-model-cache\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.090653 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.090604    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-tmp-dir\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.191140 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.191088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvxmb\" (UniqueName: \"kubernetes.io/projected/073698f5-b519-4201-9be8-d06e6f69220c-kube-api-access-rvxmb\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.191385 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.191197    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-dshm\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.191385 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.191230    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-home\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.191385 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.191250    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/073698f5-b519-4201-9be8-d06e6f69220c-tls-certs\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.191385 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.191304    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-model-cache\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.191385 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.191327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-tmp-dir\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.191849 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.191823    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-home\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.191909 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.191849    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-model-cache\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.191909 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.191889    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-tmp-dir\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.194400 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.194376    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-dshm\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.194777 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.194758    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/073698f5-b519-4201-9be8-d06e6f69220c-tls-certs\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.200421 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.200392    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvxmb\" (UniqueName: \"kubernetes.io/projected/073698f5-b519-4201-9be8-d06e6f69220c-kube-api-access-rvxmb\") pod \"e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") " pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.329013 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.328967    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:42.611808 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.611763    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"]
Sep 16 12:44:42.613368 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:44:42.613333    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod073698f5_b519_4201_9be8_d06e6f69220c.slice/crio-91c3ab4fc07f5e16578252bb68be555e96fbf883d19b10cac1f7a76c925411f1 WatchSource:0}: Error finding container 91c3ab4fc07f5e16578252bb68be555e96fbf883d19b10cac1f7a76c925411f1: Status 404 returned error can't find the container with id 91c3ab4fc07f5e16578252bb68be555e96fbf883d19b10cac1f7a76c925411f1
Sep 16 12:44:42.775354 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.775321    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88" event={"ID":"073698f5-b519-4201-9be8-d06e6f69220c","Type":"ContainerStarted","Data":"a79fc00e4f848e446d888c062b7f3433395b9ded8a03c316eb9ae70d35608a71"}
Sep 16 12:44:42.775545 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.775366    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88" event={"ID":"073698f5-b519-4201-9be8-d06e6f69220c","Type":"ContainerStarted","Data":"91c3ab4fc07f5e16578252bb68be555e96fbf883d19b10cac1f7a76c925411f1"}
Sep 16 12:44:42.793423 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:42.793375    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88" podStartSLOduration=1.7933581360000002 podStartE2EDuration="1.793358136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:44:42.791343534 +0000 UTC m=+2642.041881087" watchObservedRunningTime="2026-09-16 12:44:42.793358136 +0000 UTC m=+2642.043895739"
Sep 16 12:44:49.027364 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:49.027326    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:49.027980 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:49.027657    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:49.040587 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:49.040562    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:49.825584 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:49.825545    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:44:52.329759 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:52.329694    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:52.329759 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:52.329764    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:52.342322 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:52.342297    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:44:52.841543 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:44:52.841513    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:45:06.563442 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:06.563408    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"]
Sep 16 12:45:07.502835 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:07.502798    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"]
Sep 16 12:45:21.583361 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.583322    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h" podUID="6a3bbfb8-d292-4006-9853-b6ac862e49ff" containerName="main" containerID="cri-o://cf14fea771e5419c6f4b70118be2cb19733d6e8d8d5bd0bf8b9ae3fb7544939b" gracePeriod=45
Sep 16 12:45:21.836820 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.836756    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:45:21.954962 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.954933    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-tmp-dir\") pod \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") "
Sep 16 12:45:21.955168 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.954999    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-dshm\") pod \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") "
Sep 16 12:45:21.955168 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.955036    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-home\") pod \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") "
Sep 16 12:45:21.955168 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.955095    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3bbfb8-d292-4006-9853-b6ac862e49ff-tls-certs\") pod \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") "
Sep 16 12:45:21.955168 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.955114    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-model-cache\") pod \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") "
Sep 16 12:45:21.955168 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.955131    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkqrd\" (UniqueName: \"kubernetes.io/projected/6a3bbfb8-d292-4006-9853-b6ac862e49ff-kube-api-access-lkqrd\") pod \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\" (UID: \"6a3bbfb8-d292-4006-9853-b6ac862e49ff\") "
Sep 16 12:45:21.955457 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.955229    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a3bbfb8-d292-4006-9853-b6ac862e49ff" (UID: "6a3bbfb8-d292-4006-9853-b6ac862e49ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:45:21.955457 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.955318    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:45:21.955457 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.955386    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-home" (OuterVolumeSpecName: "home") pod "6a3bbfb8-d292-4006-9853-b6ac862e49ff" (UID: "6a3bbfb8-d292-4006-9853-b6ac862e49ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:45:21.955457 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.955393    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a3bbfb8-d292-4006-9853-b6ac862e49ff" (UID: "6a3bbfb8-d292-4006-9853-b6ac862e49ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:45:21.957468 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.957438    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-dshm" (OuterVolumeSpecName: "dshm") pod "6a3bbfb8-d292-4006-9853-b6ac862e49ff" (UID: "6a3bbfb8-d292-4006-9853-b6ac862e49ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:45:21.957592 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.957490    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a3bbfb8-d292-4006-9853-b6ac862e49ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a3bbfb8-d292-4006-9853-b6ac862e49ff" (UID: "6a3bbfb8-d292-4006-9853-b6ac862e49ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:45:21.957592 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.957497    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3bbfb8-d292-4006-9853-b6ac862e49ff-kube-api-access-lkqrd" (OuterVolumeSpecName: "kube-api-access-lkqrd") pod "6a3bbfb8-d292-4006-9853-b6ac862e49ff" (UID: "6a3bbfb8-d292-4006-9853-b6ac862e49ff"). InnerVolumeSpecName "kube-api-access-lkqrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:45:21.990974 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.990942    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a3bbfb8-d292-4006-9853-b6ac862e49ff" containerID="cf14fea771e5419c6f4b70118be2cb19733d6e8d8d5bd0bf8b9ae3fb7544939b" exitCode=0
Sep 16 12:45:21.991113 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.990986    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"
Sep 16 12:45:21.991113 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.991038    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h" event={"ID":"6a3bbfb8-d292-4006-9853-b6ac862e49ff","Type":"ContainerDied","Data":"cf14fea771e5419c6f4b70118be2cb19733d6e8d8d5bd0bf8b9ae3fb7544939b"}
Sep 16 12:45:21.991113 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.991076    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h" event={"ID":"6a3bbfb8-d292-4006-9853-b6ac862e49ff","Type":"ContainerDied","Data":"a398e742f76c16b9d89ff1daeb4a2af265c123756a2f15f01d3d44cfc0af71f1"}
Sep 16 12:45:21.991113 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:21.991092    2582 scope.go:117] "RemoveContainer" containerID="cf14fea771e5419c6f4b70118be2cb19733d6e8d8d5bd0bf8b9ae3fb7544939b"
Sep 16 12:45:22.002474 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.002443    2582 scope.go:117] "RemoveContainer" containerID="cf14fea771e5419c6f4b70118be2cb19733d6e8d8d5bd0bf8b9ae3fb7544939b"
Sep 16 12:45:22.002931 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:45:22.002908    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf14fea771e5419c6f4b70118be2cb19733d6e8d8d5bd0bf8b9ae3fb7544939b\": container with ID starting with cf14fea771e5419c6f4b70118be2cb19733d6e8d8d5bd0bf8b9ae3fb7544939b not found: ID does not exist" containerID="cf14fea771e5419c6f4b70118be2cb19733d6e8d8d5bd0bf8b9ae3fb7544939b"
Sep 16 12:45:22.003035 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.002939    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf14fea771e5419c6f4b70118be2cb19733d6e8d8d5bd0bf8b9ae3fb7544939b"} err="failed to get container status \"cf14fea771e5419c6f4b70118be2cb19733d6e8d8d5bd0bf8b9ae3fb7544939b\": rpc error: code = NotFound desc = could not find container \"cf14fea771e5419c6f4b70118be2cb19733d6e8d8d5bd0bf8b9ae3fb7544939b\": container with ID starting with cf14fea771e5419c6f4b70118be2cb19733d6e8d8d5bd0bf8b9ae3fb7544939b not found: ID does not exist"
Sep 16 12:45:22.013826 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.013801    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"]
Sep 16 12:45:22.015978 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.015957    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e14ade35-kserve-5587cc4d47-rj92h"]
Sep 16 12:45:22.056774 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.056721    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3bbfb8-d292-4006-9853-b6ac862e49ff-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:45:22.056774 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.056773    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:45:22.056947 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.056787    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkqrd\" (UniqueName: \"kubernetes.io/projected/6a3bbfb8-d292-4006-9853-b6ac862e49ff-kube-api-access-lkqrd\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:45:22.056947 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.056799    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:45:22.056947 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.056809    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a3bbfb8-d292-4006-9853-b6ac862e49ff-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:45:22.523168 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.523127    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88" podUID="073698f5-b519-4201-9be8-d06e6f69220c" containerName="main" containerID="cri-o://a79fc00e4f848e446d888c062b7f3433395b9ded8a03c316eb9ae70d35608a71" gracePeriod=45
Sep 16 12:45:22.774877 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.774808    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:45:22.964879 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.964851    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-home\") pod \"073698f5-b519-4201-9be8-d06e6f69220c\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") "
Sep 16 12:45:22.965059 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.964899    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/073698f5-b519-4201-9be8-d06e6f69220c-tls-certs\") pod \"073698f5-b519-4201-9be8-d06e6f69220c\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") "
Sep 16 12:45:22.965059 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.964973    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-model-cache\") pod \"073698f5-b519-4201-9be8-d06e6f69220c\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") "
Sep 16 12:45:22.965059 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.964992    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-dshm\") pod \"073698f5-b519-4201-9be8-d06e6f69220c\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") "
Sep 16 12:45:22.965247 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.965111    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvxmb\" (UniqueName: \"kubernetes.io/projected/073698f5-b519-4201-9be8-d06e6f69220c-kube-api-access-rvxmb\") pod \"073698f5-b519-4201-9be8-d06e6f69220c\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") "
Sep 16 12:45:22.965247 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.965122    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-home" (OuterVolumeSpecName: "home") pod "073698f5-b519-4201-9be8-d06e6f69220c" (UID: "073698f5-b519-4201-9be8-d06e6f69220c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:45:22.965247 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.965152    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-tmp-dir\") pod \"073698f5-b519-4201-9be8-d06e6f69220c\" (UID: \"073698f5-b519-4201-9be8-d06e6f69220c\") "
Sep 16 12:45:22.965438 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.965247    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-model-cache" (OuterVolumeSpecName: "model-cache") pod "073698f5-b519-4201-9be8-d06e6f69220c" (UID: "073698f5-b519-4201-9be8-d06e6f69220c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:45:22.965438 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.965407    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "073698f5-b519-4201-9be8-d06e6f69220c" (UID: "073698f5-b519-4201-9be8-d06e6f69220c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:45:22.965543 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.965528    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:45:22.965582 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.965547    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:45:22.965582 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.965555    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:45:22.967283 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.967265    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/073698f5-b519-4201-9be8-d06e6f69220c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "073698f5-b519-4201-9be8-d06e6f69220c" (UID: "073698f5-b519-4201-9be8-d06e6f69220c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:45:22.967897 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.967800    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-dshm" (OuterVolumeSpecName: "dshm") pod "073698f5-b519-4201-9be8-d06e6f69220c" (UID: "073698f5-b519-4201-9be8-d06e6f69220c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:45:22.967897 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.967872    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073698f5-b519-4201-9be8-d06e6f69220c-kube-api-access-rvxmb" (OuterVolumeSpecName: "kube-api-access-rvxmb") pod "073698f5-b519-4201-9be8-d06e6f69220c" (UID: "073698f5-b519-4201-9be8-d06e6f69220c"). InnerVolumeSpecName "kube-api-access-rvxmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:45:22.998919 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.998882    2582 generic.go:356] "Generic (PLEG): container finished" podID="073698f5-b519-4201-9be8-d06e6f69220c" containerID="a79fc00e4f848e446d888c062b7f3433395b9ded8a03c316eb9ae70d35608a71" exitCode=0
Sep 16 12:45:22.999081 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.998929    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"
Sep 16 12:45:22.999081 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.998931    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88" event={"ID":"073698f5-b519-4201-9be8-d06e6f69220c","Type":"ContainerDied","Data":"a79fc00e4f848e446d888c062b7f3433395b9ded8a03c316eb9ae70d35608a71"}
Sep 16 12:45:22.999081 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.999046    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88" event={"ID":"073698f5-b519-4201-9be8-d06e6f69220c","Type":"ContainerDied","Data":"91c3ab4fc07f5e16578252bb68be555e96fbf883d19b10cac1f7a76c925411f1"}
Sep 16 12:45:22.999081 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:22.999066    2582 scope.go:117] "RemoveContainer" containerID="a79fc00e4f848e446d888c062b7f3433395b9ded8a03c316eb9ae70d35608a71"
Sep 16 12:45:23.008714 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:23.008694    2582 scope.go:117] "RemoveContainer" containerID="a79fc00e4f848e446d888c062b7f3433395b9ded8a03c316eb9ae70d35608a71"
Sep 16 12:45:23.009043 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:45:23.009020    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79fc00e4f848e446d888c062b7f3433395b9ded8a03c316eb9ae70d35608a71\": container with ID starting with a79fc00e4f848e446d888c062b7f3433395b9ded8a03c316eb9ae70d35608a71 not found: ID does not exist" containerID="a79fc00e4f848e446d888c062b7f3433395b9ded8a03c316eb9ae70d35608a71"
Sep 16 12:45:23.009116 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:23.009057    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79fc00e4f848e446d888c062b7f3433395b9ded8a03c316eb9ae70d35608a71"} err="failed to get container status \"a79fc00e4f848e446d888c062b7f3433395b9ded8a03c316eb9ae70d35608a71\": rpc error: code = NotFound desc = could not find container \"a79fc00e4f848e446d888c062b7f3433395b9ded8a03c316eb9ae70d35608a71\": container with ID starting with a79fc00e4f848e446d888c062b7f3433395b9ded8a03c316eb9ae70d35608a71 not found: ID does not exist"
Sep 16 12:45:23.020333 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:23.020305    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"]
Sep 16 12:45:23.026286 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:23.026228    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e14ade35-kserve-58b94bf799-ftv88"]
Sep 16 12:45:23.066898 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:23.066868    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/073698f5-b519-4201-9be8-d06e6f69220c-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:45:23.066898 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:23.066900    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/073698f5-b519-4201-9be8-d06e6f69220c-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:45:23.067055 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:23.066910    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvxmb\" (UniqueName: \"kubernetes.io/projected/073698f5-b519-4201-9be8-d06e6f69220c-kube-api-access-rvxmb\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:45:23.213622 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:23.213592    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="073698f5-b519-4201-9be8-d06e6f69220c" path="/var/lib/kubelet/pods/073698f5-b519-4201-9be8-d06e6f69220c/volumes"
Sep 16 12:45:23.214020 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:23.214007    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a3bbfb8-d292-4006-9853-b6ac862e49ff" path="/var/lib/kubelet/pods/6a3bbfb8-d292-4006-9853-b6ac862e49ff/volumes"
Sep 16 12:45:27.014200 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.014165    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"]
Sep 16 12:45:27.014678 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.014524    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a3bbfb8-d292-4006-9853-b6ac862e49ff" containerName="main"
Sep 16 12:45:27.014678 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.014535    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a3bbfb8-d292-4006-9853-b6ac862e49ff" containerName="main"
Sep 16 12:45:27.014678 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.014550    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="073698f5-b519-4201-9be8-d06e6f69220c" containerName="main"
Sep 16 12:45:27.014678 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.014555    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="073698f5-b519-4201-9be8-d06e6f69220c" containerName="main"
Sep 16 12:45:27.014678 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.014607    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a3bbfb8-d292-4006-9853-b6ac862e49ff" containerName="main"
Sep 16 12:45:27.014678 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.014618    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="073698f5-b519-4201-9be8-d06e6f69220c" containerName="main"
Sep 16 12:45:27.019127 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.019103    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"
Sep 16 12:45:27.034213 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.034179    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"]
Sep 16 12:45:27.104025 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.103986    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq2ck\" (UniqueName: \"kubernetes.io/projected/7b08c314-f257-495c-a479-132b26a583d2-kube-api-access-gq2ck\") pod \"maas-api-64757c48b8-hfsxp\" (UID: \"7b08c314-f257-495c-a479-132b26a583d2\") " pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"
Sep 16 12:45:27.104025 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.104030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b08c314-f257-495c-a479-132b26a583d2-metrics-certs\") pod \"maas-api-64757c48b8-hfsxp\" (UID: \"7b08c314-f257-495c-a479-132b26a583d2\") " pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"
Sep 16 12:45:27.104307 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.104146    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b08c314-f257-495c-a479-132b26a583d2-openshift-service-ca\") pod \"maas-api-64757c48b8-hfsxp\" (UID: \"7b08c314-f257-495c-a479-132b26a583d2\") " pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"
Sep 16 12:45:27.104307 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.104190    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b08c314-f257-495c-a479-132b26a583d2-maas-api-tls\") pod \"maas-api-64757c48b8-hfsxp\" (UID: \"7b08c314-f257-495c-a479-132b26a583d2\") " pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"
Sep 16 12:45:27.205165 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.205129    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b08c314-f257-495c-a479-132b26a583d2-openshift-service-ca\") pod \"maas-api-64757c48b8-hfsxp\" (UID: \"7b08c314-f257-495c-a479-132b26a583d2\") " pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"
Sep 16 12:45:27.205295 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.205180    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b08c314-f257-495c-a479-132b26a583d2-maas-api-tls\") pod \"maas-api-64757c48b8-hfsxp\" (UID: \"7b08c314-f257-495c-a479-132b26a583d2\") " pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"
Sep 16 12:45:27.205295 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.205222    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq2ck\" (UniqueName: \"kubernetes.io/projected/7b08c314-f257-495c-a479-132b26a583d2-kube-api-access-gq2ck\") pod \"maas-api-64757c48b8-hfsxp\" (UID: \"7b08c314-f257-495c-a479-132b26a583d2\") " pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"
Sep 16 12:45:27.205295 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.205240    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b08c314-f257-495c-a479-132b26a583d2-metrics-certs\") pod \"maas-api-64757c48b8-hfsxp\" (UID: \"7b08c314-f257-495c-a479-132b26a583d2\") " pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"
Sep 16 12:45:27.205775 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.205725    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b08c314-f257-495c-a479-132b26a583d2-openshift-service-ca\") pod \"maas-api-64757c48b8-hfsxp\" (UID: \"7b08c314-f257-495c-a479-132b26a583d2\") " pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"
Sep 16 12:45:27.208144 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.208120    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b08c314-f257-495c-a479-132b26a583d2-metrics-certs\") pod \"maas-api-64757c48b8-hfsxp\" (UID: \"7b08c314-f257-495c-a479-132b26a583d2\") " pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"
Sep 16 12:45:27.208258 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.208148    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b08c314-f257-495c-a479-132b26a583d2-maas-api-tls\") pod \"maas-api-64757c48b8-hfsxp\" (UID: \"7b08c314-f257-495c-a479-132b26a583d2\") " pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"
Sep 16 12:45:27.214229 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.214207    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq2ck\" (UniqueName: \"kubernetes.io/projected/7b08c314-f257-495c-a479-132b26a583d2-kube-api-access-gq2ck\") pod \"maas-api-64757c48b8-hfsxp\" (UID: \"7b08c314-f257-495c-a479-132b26a583d2\") " pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"
Sep 16 12:45:27.332219 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.332184    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"
Sep 16 12:45:27.610963 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.610937    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"]
Sep 16 12:45:27.613494 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:45:27.613459    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b08c314_f257_495c_a479_132b26a583d2.slice/crio-dfd12dc52fc479a6d14be6ae09c9816ee218e002cbf3a6efb758c2a900a72a27 WatchSource:0}: Error finding container dfd12dc52fc479a6d14be6ae09c9816ee218e002cbf3a6efb758c2a900a72a27: Status 404 returned error can't find the container with id dfd12dc52fc479a6d14be6ae09c9816ee218e002cbf3a6efb758c2a900a72a27
Sep 16 12:45:27.615021 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:27.614999    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:45:28.030530 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:28.030435    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp" event={"ID":"7b08c314-f257-495c-a479-132b26a583d2","Type":"ContainerStarted","Data":"dfd12dc52fc479a6d14be6ae09c9816ee218e002cbf3a6efb758c2a900a72a27"}
Sep 16 12:45:29.037207 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:29.037166    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp" event={"ID":"7b08c314-f257-495c-a479-132b26a583d2","Type":"ContainerStarted","Data":"8318f52e12859ed11a418cb464188e4110552f3147fc2633090ad068102cd110"}
Sep 16 12:45:29.037638 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:29.037293    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"
Sep 16 12:45:29.056821 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:29.056770    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp" podStartSLOduration=2.470712351 podStartE2EDuration="3.056754759s" totalImagesPullingTime="586.042408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:45:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:45:29.055787994 +0000 UTC m=+2688.306325537" watchObservedRunningTime="2026-09-16 12:45:29.056754759 +0000 UTC m=+2688.307292298"
Sep 16 12:45:35.048370 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:35.048296    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-64757c48b8-hfsxp"
Sep 16 12:45:42.552597 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:42.552559    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"]
Sep 16 12:45:42.553125 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:42.552896    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx" podUID="3499149c-92a9-4514-8dfb-f57bcd1b66b0" containerName="maas-api" containerID="cri-o://d79da1d96e8de25317fdec5b086fbbf15b9e9f1d276934892a4208731502bc45" gracePeriod=30
Sep 16 12:45:42.946420 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:42.946375    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx" podUID="3499149c-92a9-4514-8dfb-f57bcd1b66b0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.45:8443/health\": dial tcp 10.133.0.45:8443: connect: connection refused"
Sep 16 12:45:43.803168 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:43.803142    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:45:43.842515 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:43.842482    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3499149c-92a9-4514-8dfb-f57bcd1b66b0-maas-api-tls\") pod \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\" (UID: \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\") "
Sep 16 12:45:43.842515 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:43.842523    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3499149c-92a9-4514-8dfb-f57bcd1b66b0-openshift-service-ca\") pod \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\" (UID: \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\") "
Sep 16 12:45:43.842789 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:43.842585    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3499149c-92a9-4514-8dfb-f57bcd1b66b0-metrics-certs\") pod \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\" (UID: \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\") "
Sep 16 12:45:43.842789 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:43.842636    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml8gx\" (UniqueName: \"kubernetes.io/projected/3499149c-92a9-4514-8dfb-f57bcd1b66b0-kube-api-access-ml8gx\") pod \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\" (UID: \"3499149c-92a9-4514-8dfb-f57bcd1b66b0\") "
Sep 16 12:45:43.843047 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:43.843014    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3499149c-92a9-4514-8dfb-f57bcd1b66b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3499149c-92a9-4514-8dfb-f57bcd1b66b0" (UID: "3499149c-92a9-4514-8dfb-f57bcd1b66b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:45:43.844984 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:43.844952    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3499149c-92a9-4514-8dfb-f57bcd1b66b0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3499149c-92a9-4514-8dfb-f57bcd1b66b0" (UID: "3499149c-92a9-4514-8dfb-f57bcd1b66b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:45:43.845104 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:43.845000    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3499149c-92a9-4514-8dfb-f57bcd1b66b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3499149c-92a9-4514-8dfb-f57bcd1b66b0" (UID: "3499149c-92a9-4514-8dfb-f57bcd1b66b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:45:43.845104 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:43.845037    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3499149c-92a9-4514-8dfb-f57bcd1b66b0-kube-api-access-ml8gx" (OuterVolumeSpecName: "kube-api-access-ml8gx") pod "3499149c-92a9-4514-8dfb-f57bcd1b66b0" (UID: "3499149c-92a9-4514-8dfb-f57bcd1b66b0"). InnerVolumeSpecName "kube-api-access-ml8gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:45:43.944036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:43.943932    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml8gx\" (UniqueName: \"kubernetes.io/projected/3499149c-92a9-4514-8dfb-f57bcd1b66b0-kube-api-access-ml8gx\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:45:43.944036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:43.943971    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3499149c-92a9-4514-8dfb-f57bcd1b66b0-maas-api-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:45:43.944036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:43.943982    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3499149c-92a9-4514-8dfb-f57bcd1b66b0-openshift-service-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:45:43.944036 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:43.943996    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3499149c-92a9-4514-8dfb-f57bcd1b66b0-metrics-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
Sep 16 12:45:44.123829 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:44.123779    2582 generic.go:356] "Generic (PLEG): container finished" podID="3499149c-92a9-4514-8dfb-f57bcd1b66b0" containerID="d79da1d96e8de25317fdec5b086fbbf15b9e9f1d276934892a4208731502bc45" exitCode=0
Sep 16 12:45:44.124025 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:44.123837    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"
Sep 16 12:45:44.124025 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:44.123866    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx" event={"ID":"3499149c-92a9-4514-8dfb-f57bcd1b66b0","Type":"ContainerDied","Data":"d79da1d96e8de25317fdec5b086fbbf15b9e9f1d276934892a4208731502bc45"}
Sep 16 12:45:44.124025 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:44.123920    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx" event={"ID":"3499149c-92a9-4514-8dfb-f57bcd1b66b0","Type":"ContainerDied","Data":"28657cdfe17de5858127d3bfc1429748f7e8a7400646f1251e254cda9b46440c"}
Sep 16 12:45:44.124025 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:44.123942    2582 scope.go:117] "RemoveContainer" containerID="d79da1d96e8de25317fdec5b086fbbf15b9e9f1d276934892a4208731502bc45"
Sep 16 12:45:44.134228 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:44.134212    2582 scope.go:117] "RemoveContainer" containerID="d79da1d96e8de25317fdec5b086fbbf15b9e9f1d276934892a4208731502bc45"
Sep 16 12:45:44.134513 ip-10-0-140-71 kubenswrapper[2582]: E0916 12:45:44.134493    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79da1d96e8de25317fdec5b086fbbf15b9e9f1d276934892a4208731502bc45\": container with ID starting with d79da1d96e8de25317fdec5b086fbbf15b9e9f1d276934892a4208731502bc45 not found: ID does not exist" containerID="d79da1d96e8de25317fdec5b086fbbf15b9e9f1d276934892a4208731502bc45"
Sep 16 12:45:44.134611 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:44.134523    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79da1d96e8de25317fdec5b086fbbf15b9e9f1d276934892a4208731502bc45"} err="failed to get container status \"d79da1d96e8de25317fdec5b086fbbf15b9e9f1d276934892a4208731502bc45\": rpc error: code = NotFound desc = could not find container \"d79da1d96e8de25317fdec5b086fbbf15b9e9f1d276934892a4208731502bc45\": container with ID starting with d79da1d96e8de25317fdec5b086fbbf15b9e9f1d276934892a4208731502bc45 not found: ID does not exist"
Sep 16 12:45:44.146379 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:44.146351    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"]
Sep 16 12:45:44.150858 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:44.150828    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-74767d9c48-vdvqx"]
Sep 16 12:45:45.212308 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:45.212273    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3499149c-92a9-4514-8dfb-f57bcd1b66b0" path="/var/lib/kubelet/pods/3499149c-92a9-4514-8dfb-f57bcd1b66b0/volumes"
Sep 16 12:45:56.833323 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:45:56.833287    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c677858cf-vqbww_fb8c8c07-9f8b-4f8c-9ceb-113c9a685e7e/authorino/0.log"
Sep 16 12:46:02.748784 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:02.748728    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7887d7f7b8-6v6jp_5eed7827-c019-4b9d-b655-9c1c36223c1d/copy-manifests/0.log"
Sep 16 12:46:02.755976 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:02.755954    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7887d7f7b8-6v6jp_5eed7827-c019-4b9d-b655-9c1c36223c1d/manager/0.log"
Sep 16 12:46:03.456342 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:03.456306    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-64757c48b8-hfsxp_7b08c314-f257-495c-a479-132b26a583d2/maas-api/0.log"
Sep 16 12:46:04.605200 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:04.605170    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c677858cf-vqbww_fb8c8c07-9f8b-4f8c-9ceb-113c9a685e7e/authorino/0.log"
Sep 16 12:46:04.943636 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:04.943552    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-x6llb_d39938d5-6891-4ee8-b7e4-85452ea78911/kuadrant-console-plugin/0.log"
Sep 16 12:46:05.906650 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:05.906605    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55f6c77cbf-64gz4_630a5bd9-f0fc-4806-9cba-aa16fd039c86/kube-auth-proxy/0.log"
Sep 16 12:46:06.479510 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:06.479470    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69bbc96588-pkg66_82d9505e-b799-4990-9927-e308c84e5292/router/0.log"
Sep 16 12:46:07.310827 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:07.310793    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgshnd_01abe45e-1cb1-44a2-b66e-a1e3520994ac/main/0.log"
Sep 16 12:46:07.548347 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:07.548319    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-m4f7f_6b29d12d-8575-43f5-8d11-15e5e85acdfc/main/0.log"
Sep 16 12:46:12.836561 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:12.836519    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mv6dr/must-gather-cbr9n"]
Sep 16 12:46:12.837085 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:12.837028    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3499149c-92a9-4514-8dfb-f57bcd1b66b0" containerName="maas-api"
Sep 16 12:46:12.837085 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:12.837048    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3499149c-92a9-4514-8dfb-f57bcd1b66b0" containerName="maas-api"
Sep 16 12:46:12.837208 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:12.837156    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3499149c-92a9-4514-8dfb-f57bcd1b66b0" containerName="maas-api"
Sep 16 12:46:12.840568 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:12.840546    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mv6dr/must-gather-cbr9n"
Sep 16 12:46:12.849855 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:12.849827    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mv6dr/must-gather-cbr9n"]
Sep 16 12:46:12.851695 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:12.851675    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mv6dr\"/\"kube-root-ca.crt\""
Sep 16 12:46:12.851825 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:12.851691    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mv6dr\"/\"openshift-service-ca.crt\""
Sep 16 12:46:12.851825 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:12.851719    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mv6dr\"/\"default-dockercfg-ww4vd\""
Sep 16 12:46:13.002797 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:13.002762    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snpkp\" (UniqueName: \"kubernetes.io/projected/a62ea2c7-bc46-4312-a86e-91f42f6e1b6d-kube-api-access-snpkp\") pod \"must-gather-cbr9n\" (UID: \"a62ea2c7-bc46-4312-a86e-91f42f6e1b6d\") " pod="openshift-must-gather-mv6dr/must-gather-cbr9n"
Sep 16 12:46:13.003002 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:13.002907    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a62ea2c7-bc46-4312-a86e-91f42f6e1b6d-must-gather-output\") pod \"must-gather-cbr9n\" (UID: \"a62ea2c7-bc46-4312-a86e-91f42f6e1b6d\") " pod="openshift-must-gather-mv6dr/must-gather-cbr9n"
Sep 16 12:46:13.104428 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:13.104325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a62ea2c7-bc46-4312-a86e-91f42f6e1b6d-must-gather-output\") pod \"must-gather-cbr9n\" (UID: \"a62ea2c7-bc46-4312-a86e-91f42f6e1b6d\") " pod="openshift-must-gather-mv6dr/must-gather-cbr9n"
Sep 16 12:46:13.104428 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:13.104382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snpkp\" (UniqueName: \"kubernetes.io/projected/a62ea2c7-bc46-4312-a86e-91f42f6e1b6d-kube-api-access-snpkp\") pod \"must-gather-cbr9n\" (UID: \"a62ea2c7-bc46-4312-a86e-91f42f6e1b6d\") " pod="openshift-must-gather-mv6dr/must-gather-cbr9n"
Sep 16 12:46:13.104718 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:13.104698    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a62ea2c7-bc46-4312-a86e-91f42f6e1b6d-must-gather-output\") pod \"must-gather-cbr9n\" (UID: \"a62ea2c7-bc46-4312-a86e-91f42f6e1b6d\") " pod="openshift-must-gather-mv6dr/must-gather-cbr9n"
Sep 16 12:46:13.126050 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:13.126024    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snpkp\" (UniqueName: \"kubernetes.io/projected/a62ea2c7-bc46-4312-a86e-91f42f6e1b6d-kube-api-access-snpkp\") pod \"must-gather-cbr9n\" (UID: \"a62ea2c7-bc46-4312-a86e-91f42f6e1b6d\") " pod="openshift-must-gather-mv6dr/must-gather-cbr9n"
Sep 16 12:46:13.151653 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:13.151614    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mv6dr/must-gather-cbr9n"
Sep 16 12:46:13.435209 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:13.435127    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mv6dr/must-gather-cbr9n"]
Sep 16 12:46:13.438501 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:46:13.438470    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda62ea2c7_bc46_4312_a86e_91f42f6e1b6d.slice/crio-0bbfd897b84afe9082b771e9ab81f96cc3a6d4352da8cd02f7976c144cda96b9 WatchSource:0}: Error finding container 0bbfd897b84afe9082b771e9ab81f96cc3a6d4352da8cd02f7976c144cda96b9: Status 404 returned error can't find the container with id 0bbfd897b84afe9082b771e9ab81f96cc3a6d4352da8cd02f7976c144cda96b9
Sep 16 12:46:14.283307 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:14.283279    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mv6dr/must-gather-cbr9n" event={"ID":"a62ea2c7-bc46-4312-a86e-91f42f6e1b6d","Type":"ContainerStarted","Data":"03b72ba3991291007d20442c1411db2815af774f884295da85ed57ce0f5b146d"}
Sep 16 12:46:14.283648 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:14.283329    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mv6dr/must-gather-cbr9n" event={"ID":"a62ea2c7-bc46-4312-a86e-91f42f6e1b6d","Type":"ContainerStarted","Data":"0bbfd897b84afe9082b771e9ab81f96cc3a6d4352da8cd02f7976c144cda96b9"}
Sep 16 12:46:15.293087 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:15.293030    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mv6dr/must-gather-cbr9n" event={"ID":"a62ea2c7-bc46-4312-a86e-91f42f6e1b6d","Type":"ContainerStarted","Data":"050aaa9d3bd0c37668c9fdd6d41b4892916a1b1a5b723e2e26540905ae690296"}
Sep 16 12:46:15.319135 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:15.319068    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mv6dr/must-gather-cbr9n" podStartSLOduration=2.5914530879999997 podStartE2EDuration="3.319043785s" totalImagesPullingTime="727.590697ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:46:15.318238095 +0000 UTC m=+2734.568775648" watchObservedRunningTime="2026-09-16 12:46:15.319043785 +0000 UTC m=+2734.569581336"
Sep 16 12:46:15.798423 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:15.798385    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8drdm_bef09744-222b-4d55-9d87-aca589bd4f13/global-pull-secret-syncer/0.log"
Sep 16 12:46:15.949807 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:15.949771    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z7wfg_c51bc19e-789f-4167-80b8-ada031cdfc55/konnectivity-agent/0.log"
Sep 16 12:46:16.028873 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:16.028836    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-71.ec2.internal_daddab1c0f05fb32f4f8c0ebec22f11c/haproxy/0.log"
Sep 16 12:46:20.415065 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:20.415027    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c677858cf-vqbww_fb8c8c07-9f8b-4f8c-9ceb-113c9a685e7e/authorino/0.log"
Sep 16 12:46:20.520166 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:20.520048    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-x6llb_d39938d5-6891-4ee8-b7e4-85452ea78911/kuadrant-console-plugin/0.log"
Sep 16 12:46:22.574203 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:22.574168    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-fmxr7_1af8115e-e839-46e5-9297-22d894d741cb/kube-state-metrics/0.log"
Sep 16 12:46:22.594179 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:22.594134    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-fmxr7_1af8115e-e839-46e5-9297-22d894d741cb/kube-rbac-proxy-main/0.log"
Sep 16 12:46:22.621110 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:22.621072    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-fmxr7_1af8115e-e839-46e5-9297-22d894d741cb/kube-rbac-proxy-self/0.log"
Sep 16 12:46:22.674807 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:22.674773    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-smw9t_7d9d891d-9a19-4bd4-b9da-ffbcefdd0a32/monitoring-plugin/0.log"
Sep 16 12:46:22.856565 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:22.856483    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w4qth_e603c80f-285b-402e-a4a4-dedc28d07e6e/node-exporter/0.log"
Sep 16 12:46:22.877673 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:22.877639    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w4qth_e603c80f-285b-402e-a4a4-dedc28d07e6e/kube-rbac-proxy/0.log"
Sep 16 12:46:22.901775 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:22.901729    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w4qth_e603c80f-285b-402e-a4a4-dedc28d07e6e/init-textfile/0.log"
Sep 16 12:46:22.935599 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:22.935564    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-zt8v8_485a8319-25f9-4f8c-a321-aae274ceaa8c/kube-rbac-proxy-main/0.log"
Sep 16 12:46:22.959201 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:22.959169    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-zt8v8_485a8319-25f9-4f8c-a321-aae274ceaa8c/kube-rbac-proxy-self/0.log"
Sep 16 12:46:22.979793 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:22.979752    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-zt8v8_485a8319-25f9-4f8c-a321-aae274ceaa8c/openshift-state-metrics/0.log"
Sep 16 12:46:23.277485 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:23.277451    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-nms9k_7dfce9d4-6d98-4673-81eb-76c55fef3ccf/prometheus-operator-admission-webhook/0.log"
Sep 16 12:46:26.153583 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.153551    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-74f74598d6-pzbjz_75583f57-60b9-4d0a-9795-de7792fea195/console/0.log"
Sep 16 12:46:26.361650 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.361602    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"]
Sep 16 12:46:26.369921 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.369895    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:26.374804 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.374773    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"]
Sep 16 12:46:26.545136 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.545035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22pbx\" (UniqueName: \"kubernetes.io/projected/fe79f81e-b0dc-419f-8ac2-a4aa57499284-kube-api-access-22pbx\") pod \"perf-node-gather-daemonset-4nl5q\" (UID: \"fe79f81e-b0dc-419f-8ac2-a4aa57499284\") " pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:26.545408 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.545382    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe79f81e-b0dc-419f-8ac2-a4aa57499284-podres\") pod \"perf-node-gather-daemonset-4nl5q\" (UID: \"fe79f81e-b0dc-419f-8ac2-a4aa57499284\") " pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:26.545549 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.545533    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe79f81e-b0dc-419f-8ac2-a4aa57499284-sys\") pod \"perf-node-gather-daemonset-4nl5q\" (UID: \"fe79f81e-b0dc-419f-8ac2-a4aa57499284\") " pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:26.545671 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.545648    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe79f81e-b0dc-419f-8ac2-a4aa57499284-lib-modules\") pod \"perf-node-gather-daemonset-4nl5q\" (UID: \"fe79f81e-b0dc-419f-8ac2-a4aa57499284\") " pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:26.545856 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.545830    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe79f81e-b0dc-419f-8ac2-a4aa57499284-proc\") pod \"perf-node-gather-daemonset-4nl5q\" (UID: \"fe79f81e-b0dc-419f-8ac2-a4aa57499284\") " pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:26.647433 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.647393    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe79f81e-b0dc-419f-8ac2-a4aa57499284-proc\") pod \"perf-node-gather-daemonset-4nl5q\" (UID: \"fe79f81e-b0dc-419f-8ac2-a4aa57499284\") " pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:26.647636 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.647485    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22pbx\" (UniqueName: \"kubernetes.io/projected/fe79f81e-b0dc-419f-8ac2-a4aa57499284-kube-api-access-22pbx\") pod \"perf-node-gather-daemonset-4nl5q\" (UID: \"fe79f81e-b0dc-419f-8ac2-a4aa57499284\") " pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:26.647636 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.647512    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe79f81e-b0dc-419f-8ac2-a4aa57499284-podres\") pod \"perf-node-gather-daemonset-4nl5q\" (UID: \"fe79f81e-b0dc-419f-8ac2-a4aa57499284\") " pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:26.647636 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.647540    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe79f81e-b0dc-419f-8ac2-a4aa57499284-sys\") pod \"perf-node-gather-daemonset-4nl5q\" (UID: \"fe79f81e-b0dc-419f-8ac2-a4aa57499284\") " pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:26.647636 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.647567    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe79f81e-b0dc-419f-8ac2-a4aa57499284-lib-modules\") pod \"perf-node-gather-daemonset-4nl5q\" (UID: \"fe79f81e-b0dc-419f-8ac2-a4aa57499284\") " pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:26.647869 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.647642    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe79f81e-b0dc-419f-8ac2-a4aa57499284-podres\") pod \"perf-node-gather-daemonset-4nl5q\" (UID: \"fe79f81e-b0dc-419f-8ac2-a4aa57499284\") " pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:26.647869 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.647538    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe79f81e-b0dc-419f-8ac2-a4aa57499284-proc\") pod \"perf-node-gather-daemonset-4nl5q\" (UID: \"fe79f81e-b0dc-419f-8ac2-a4aa57499284\") " pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:26.647869 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.647661    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe79f81e-b0dc-419f-8ac2-a4aa57499284-sys\") pod \"perf-node-gather-daemonset-4nl5q\" (UID: \"fe79f81e-b0dc-419f-8ac2-a4aa57499284\") " pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:26.647869 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.647723    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe79f81e-b0dc-419f-8ac2-a4aa57499284-lib-modules\") pod \"perf-node-gather-daemonset-4nl5q\" (UID: \"fe79f81e-b0dc-419f-8ac2-a4aa57499284\") " pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:26.656778 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.656723    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22pbx\" (UniqueName: \"kubernetes.io/projected/fe79f81e-b0dc-419f-8ac2-a4aa57499284-kube-api-access-22pbx\") pod \"perf-node-gather-daemonset-4nl5q\" (UID: \"fe79f81e-b0dc-419f-8ac2-a4aa57499284\") " pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:26.684435 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.684397    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:26.984017 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:26.983984    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"]
Sep 16 12:46:26.984913 ip-10-0-140-71 kubenswrapper[2582]: W0916 12:46:26.984877    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfe79f81e_b0dc_419f_8ac2_a4aa57499284.slice/crio-839ae6222ff08c7e67106e26f786249f8976077d41242a3edd536b7573bd5234 WatchSource:0}: Error finding container 839ae6222ff08c7e67106e26f786249f8976077d41242a3edd536b7573bd5234: Status 404 returned error can't find the container with id 839ae6222ff08c7e67106e26f786249f8976077d41242a3edd536b7573bd5234
Sep 16 12:46:27.010857 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:27.010827    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-24dfb_b7b80a25-50dd-45f1-8118-23adabe85425/volume-data-source-validator/0.log"
Sep 16 12:46:27.395923 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:27.395873    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q" event={"ID":"fe79f81e-b0dc-419f-8ac2-a4aa57499284","Type":"ContainerStarted","Data":"996d1ff80e09d6276d14d597804b2d51d70bc50256ba613dae7b56b04b0ad191"}
Sep 16 12:46:27.396405 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:27.395957    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q" event={"ID":"fe79f81e-b0dc-419f-8ac2-a4aa57499284","Type":"ContainerStarted","Data":"839ae6222ff08c7e67106e26f786249f8976077d41242a3edd536b7573bd5234"}
Sep 16 12:46:27.396405 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:27.396097    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:28.277510 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:28.277481    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wf5rh_aab5b1d1-1603-47a0-a5cc-0d136c7f678a/dns/0.log"
Sep 16 12:46:28.301533 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:28.301506    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wf5rh_aab5b1d1-1603-47a0-a5cc-0d136c7f678a/kube-rbac-proxy/0.log"
Sep 16 12:46:28.327508 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:28.327474    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4hzq6_e5308201-4e49-461a-be86-4c661c0d2783/dns-node-resolver/0.log"
Sep 16 12:46:28.964830 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:28.964793    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7q8hz_55b45601-95e3-4427-af82-2358088e7e8a/node-ca/0.log"
Sep 16 12:46:30.342151 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:30.342119    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55f6c77cbf-64gz4_630a5bd9-f0fc-4806-9cba-aa16fd039c86/kube-auth-proxy/0.log"
Sep 16 12:46:30.479427 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:30.479382    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69bbc96588-pkg66_82d9505e-b799-4990-9927-e308c84e5292/router/0.log"
Sep 16 12:46:31.412959 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:31.412922    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t8wnw_84e4fef2-7c44-4c75-a75a-20fc52336320/serve-healthcheck-canary/0.log"
Sep 16 12:46:31.894574 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:31.894540    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-9hpds_747ed2fa-e045-4333-a75c-20d76a36ecf6/insights-operator/0.log"
Sep 16 12:46:31.895200 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:31.895173    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-9hpds_747ed2fa-e045-4333-a75c-20d76a36ecf6/insights-operator/1.log"
Sep 16 12:46:31.918131 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:31.918103    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-57ksj_59d5b11b-1cb0-411d-bc9a-b16ff5a304c2/kube-rbac-proxy/0.log"
Sep 16 12:46:31.936276 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:31.936240    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-57ksj_59d5b11b-1cb0-411d-bc9a-b16ff5a304c2/exporter/0.log"
Sep 16 12:46:31.954775 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:31.954729    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-57ksj_59d5b11b-1cb0-411d-bc9a-b16ff5a304c2/extractor/0.log"
Sep 16 12:46:33.420593 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:33.419581    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q"
Sep 16 12:46:33.442617 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:33.442562    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mv6dr/perf-node-gather-daemonset-4nl5q" podStartSLOduration=7.442546996 podStartE2EDuration="7.442546996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:46:27.416645225 +0000 UTC m=+2746.667182774" watchObservedRunningTime="2026-09-16 12:46:33.442546996 +0000 UTC m=+2752.693084589"
Sep 16 12:46:34.625765 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:34.625708    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7887d7f7b8-6v6jp_5eed7827-c019-4b9d-b655-9c1c36223c1d/manager/0.log"
Sep 16 12:46:34.647364 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:34.647339    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7887d7f7b8-6v6jp_5eed7827-c019-4b9d-b655-9c1c36223c1d/copy-manifests/0.log"
Sep 16 12:46:36.265755 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:36.265704    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-c7f7d6d7f-5dvlv_93e1607f-5197-4a64-ae99-9de6a620763e/manager/0.log"
Sep 16 12:46:37.336931 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:37.336901    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55f6c77cbf-64gz4_630a5bd9-f0fc-4806-9cba-aa16fd039c86/kube-auth-proxy/0.log"
Sep 16 12:46:37.449838 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:37.449809    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69bbc96588-pkg66_82d9505e-b799-4990-9927-e308c84e5292/router/0.log"
Sep 16 12:46:39.520750 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:39.520700    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55f6c77cbf-64gz4_630a5bd9-f0fc-4806-9cba-aa16fd039c86/kube-auth-proxy/0.log"
Sep 16 12:46:39.627069 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:39.627032    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69bbc96588-pkg66_82d9505e-b799-4990-9927-e308c84e5292/router/0.log"
Sep 16 12:46:43.031278 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:43.031232    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5smzb_3fc4ace4-ea76-4d43-8722-093f76f186dd/kube-multus/0.log"
Sep 16 12:46:43.201487 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:43.201460    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w69gx_25085fb2-c567-4b20-bd8d-6241d7cce1cf/kube-multus-additional-cni-plugins/0.log"
Sep 16 12:46:43.222001 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:43.221969    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w69gx_25085fb2-c567-4b20-bd8d-6241d7cce1cf/egress-router-binary-copy/0.log"
Sep 16 12:46:43.240446 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:43.240415    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w69gx_25085fb2-c567-4b20-bd8d-6241d7cce1cf/cni-plugins/0.log"
Sep 16 12:46:43.259344 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:43.259319    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w69gx_25085fb2-c567-4b20-bd8d-6241d7cce1cf/bond-cni-plugin/0.log"
Sep 16 12:46:43.277157 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:43.277101    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w69gx_25085fb2-c567-4b20-bd8d-6241d7cce1cf/routeoverride-cni/0.log"
Sep 16 12:46:43.294864 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:43.294783    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w69gx_25085fb2-c567-4b20-bd8d-6241d7cce1cf/whereabouts-cni-bincopy/0.log"
Sep 16 12:46:43.313088 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:43.313058    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w69gx_25085fb2-c567-4b20-bd8d-6241d7cce1cf/whereabouts-cni/0.log"
Sep 16 12:46:43.625195 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:43.625164    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bflx7_9a90e5c3-5fcb-4016-8682-3803c3121620/network-metrics-daemon/0.log"
Sep 16 12:46:43.641299 ip-10-0-140-71 kubenswrapper[2582]: I0916 12:46:43.641273    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bflx7_9a90e5c3-5fcb-4016-8682-3803c3121620/kube-rbac-proxy/0.log"